From 8b32433bd42c07baeaa3ee482f8b2138f8957270 Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Thu, 27 Mar 2025 12:49:53 -0700 Subject: [PATCH 01/36] Implement models for Neon MQ API (#18) * Add neon MQ request models and update MQ validator for Klat Observer compat * Add validator to troubleshoot missing `sid` data * Troubleshooting utterance parsing * Troubleshoot expected data models * Fix logical error in MQ Context handling * Add missing `lang` data Troubleshooting Messagebus connector errors * Enforce a default `timing` context as unsafe references exist in the `neon-messagebus-mq-connector` module * Implement Timing Context for SIO Klat messages * Dump timing contexts to float values for serialization support and to troubleshoot errors in `messagebus-mq-connector` * Move `model_dump` override to `BaseModel` so that all models are guranteed serializable * Fix logical error in context validator changes (allow empty string values) * Update global `model_dump` per Copilot to recurse through dicts * Define `session` context for Klat messages. Use `cid` as `session_id` so that each Klat conversation is mapped to a unique session * Add `types` module to define `Literal` definitions Define `models.api.messagebus` module with Serialized `Message` definitions and unit test coverage Update base `MessageContext` definition to include a default `session` value as there are unsafe references found in the wild * Update `models.api.mq.neon` to extend base message definitions Normalize class names to CamelCase Refactor `NeonApiMessage` to implement a `TypeAdaptor`, similar to Users CRUD API Refactor `test_mq` and add coverage for added Neon MQ API models * Add module-level import for `models.api.messagebus` submodule * Add `NeonAudioInput` message and refactor `NeonGetResponse to `NeonTextInput` for naming consistency * Refactor MQ message to `NeonMqTextInput` for consistency Add `NeonMqAudioInput` model Move invalid model validation to `NeonApiMessage.__new__` Include Claude-written unit tests * Add `BaseMessage.as_message` to easily export a Pydantic object as a `Message` * Handle Pydantic models in message data * Add validator to `BaseMessage` to handle serializing Message objects * Undo non-working BaseMessage validation * Refactor `gender` type to `Gender` for consistency with other types Fix definition error in `TtsResponseData` with updated unit tests * Fix declared type error in `context.session.active_skills` annotation * Handle `None` phonemes with unit test coverage * Mark `phonemes` as optional to account for current implementation * Cleanup unit tests to resolve failures * Implement Language message support for Iris * Explicitly handle placeholder DoB with Claude-written unit test coverage * Handle input string for `destination` context with unit test coverage * Implement NeonMqUnknownMessage to allow arbitrary messages to be passed to MQ-connected services for backwards-compat. Implement test coverage by Claude * Add `male` and `female` fields to `TtsResponse` with unit tests * Remove `print` line which will pollute logs * Add default Klat context SID/CID for backwards-compat. with IRIS requests * Update `api.messagebus` module to include `Field` values for all model params Update `api.test_messagebus` test module per Claude suggestions Add Claude-generated validators to models in `api.messagebus` to better validate inputs * Add comments to validator overrides * Add docstrings to `models.api.mq` added classes Add top-level import of new `mq` submodule Add missing model class to `__all__` * Add docstrings to `NeonApiMessage` * Enforce UTC timezone in TimingContext for parsed timestamps * Add field annotations to KlatContext * Implement more Claude-written tests of timing context * Add test case and annotation for `as_message` method Add `ovos-bus-client` to test requirements * Add `destination` context validation for compat. with neon-messagebus-mq-connector https://github.com/NeonGeckoCom/neon-messagebus-mq-connector/actions/runs/14047562398/job/39331567715?pr=61 * Update API message context handling to override default `destination` for `audio` messages * Resolve `messagebus` editor warning * Update `KlatContext` default value handling to resolve backwards-compat. issue in `pyklatchat-observer` with update unit test * Address review feedback * Minor code style formatting change * Remove license_tests package overrides to troubleshoot GHA failure * Whitelist typing_extensions with PSF license * Add validator for `session` context handling backwards-compat with test coverage * Update license notices to 2025 --- .github/workflows/license_tests.yml | 2 +- LICENSE.md | 2 +- neon_data_models/__init__.py | 2 +- neon_data_models/enum.py | 2 +- neon_data_models/models/__init__.py | 2 +- neon_data_models/models/api/__init__.py | 3 +- neon_data_models/models/api/jwt.py | 2 +- neon_data_models/models/api/llm.py | 2 +- .../models/api/messagebus/__init__.py | 230 ++++++ neon_data_models/models/api/mq/__init__.py | 3 +- neon_data_models/models/api/mq/llm.py | 2 +- neon_data_models/models/api/mq/neon.py | 197 +++++ neon_data_models/models/api/mq/users.py | 2 +- .../models/api/node_v1/__init__.py | 2 +- neon_data_models/models/base/__init__.py | 36 +- neon_data_models/models/base/contexts.py | 65 +- neon_data_models/models/base/messagebus.py | 45 +- neon_data_models/models/client/__init__.py | 2 +- neon_data_models/models/client/node.py | 2 +- neon_data_models/models/user/__init__.py | 2 +- neon_data_models/models/user/database.py | 14 +- neon_data_models/models/user/neon_profile.py | 2 +- neon_data_models/types.py | 30 + neon_data_models/util.py | 2 +- neon_data_models/version.py | 2 +- requirements/test.txt | 3 +- setup.py | 2 +- tests/models/api/test_llm.py | 2 +- tests/models/api/test_messagebus.py | 719 ++++++++++++++++++ tests/models/api/test_mq.py | 526 ++++++++++++- tests/models/api/test_node.py | 2 +- tests/models/test_base.py | 302 +++++++- tests/models/test_client.py | 2 +- tests/models/test_user.py | 62 +- tests/test_enum.py | 2 +- tests/test_imports.py | 2 +- 36 files changed, 2223 insertions(+), 56 deletions(-) create mode 100644 neon_data_models/models/api/messagebus/__init__.py create mode 100644 neon_data_models/models/api/mq/neon.py create mode 100644 neon_data_models/types.py create mode 100644 tests/models/api/test_messagebus.py diff --git a/.github/workflows/license_tests.yml b/.github/workflows/license_tests.yml index 6285b19f..9469733c 100644 --- a/.github/workflows/license_tests.yml +++ b/.github/workflows/license_tests.yml @@ -9,4 +9,4 @@ jobs: license_tests: uses: neongeckocom/.github/.github/workflows/license_tests.yml@master with: - packages-exclude: '^(neon-data-models|dnspython).*' + packages-exclude: '^(neon-data-models|dnspython|typing_extensions).*' diff --git a/LICENSE.md b/LICENSE.md index 17981f0a..1072bd39 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 Redistribution and use in source and binary forms, with or without modification, are permitted provided that the diff --git a/neon_data_models/__init__.py b/neon_data_models/__init__.py index 8d9d8f6f..f6f252ae 100644 --- a/neon_data_models/__init__.py +++ b/neon_data_models/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/enum.py b/neon_data_models/enum.py index f5063079..d228f1ae 100644 --- a/neon_data_models/enum.py +++ b/neon_data_models/enum.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/__init__.py b/neon_data_models/models/__init__.py index 3ca2d288..e6cd0063 100644 --- a/neon_data_models/models/__init__.py +++ b/neon_data_models/models/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/__init__.py b/neon_data_models/models/api/__init__.py index ba7a6af8..e0e9c2bb 100644 --- a/neon_data_models/models/api/__init__.py +++ b/neon_data_models/models/api/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -28,3 +28,4 @@ from neon_data_models.models.api.jwt import * from neon_data_models.models.api.llm import * from neon_data_models.models.api.mq import * +from neon_data_models.models.api.messagebus import * diff --git a/neon_data_models/models/api/jwt.py b/neon_data_models/models/api/jwt.py index 61680a38..f6ef9000 100644 --- a/neon_data_models/models/api/jwt.py +++ b/neon_data_models/models/api/jwt.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/llm.py b/neon_data_models/models/api/llm.py index abb8f537..a967f462 100644 --- a/neon_data_models/models/api/llm.py +++ b/neon_data_models/models/api/llm.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/messagebus/__init__.py b/neon_data_models/models/api/messagebus/__init__.py new file mode 100644 index 00000000..2b4a6cb0 --- /dev/null +++ b/neon_data_models/models/api/messagebus/__init__.py @@ -0,0 +1,230 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2025 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +""" +Defines models for `Message` objects sent on the Neon messagebus. +""" + +from typing import Any, List, Literal, Dict, Optional +from pydantic import Field, model_validator + +from neon_data_models.types import Gender +from neon_data_models.models.base import BaseModel +from neon_data_models.models.base.messagebus import BaseMessage + + +# Data models +class GetTtsData(BaseModel): + text: str = Field(description="Text to be spoken") + lang: str = Field(default="en-us", + description="BCP-47 language code for TTS") + + @model_validator(mode='before') + @classmethod + def validate_inputs(cls, values): + # Ensure input values are normalized to valid field names + if hasattr(values, 'model_dump'): + values = values.model_dump() + if 'text' not in values: + values['text'] = values.get('utterance') + return values + + +class TtsResponse(BaseModel): + sentence: str = Field(description="Text to be spoken") + translated: bool = Field( + default=False, + description="True if the text has been translated before TTS synthesis") + phonemes: Optional[str] = Field( + default=None, description="Phoneme representation of the sentence") + genders: List[Gender] = Field( + description="List of genders included in the response `audio` field", + deprecated=True) + male: Optional[str] = Field(default=None, + description="Path to audio file in male voice") + female: Optional[str] = Field( + default=None, description="Path to audio file in female voice") + audio: Dict[Gender, str] + + @model_validator(mode='after') + def validate_gender_audio_match(self): + """Validate that the genders list matches the audio dictionary keys.""" + if not set(self.genders) == set(self.audio.keys()): + raise ValueError( + "All genders listed in 'genders' must have corresponding " + "entries in 'audio'") + return self + + +class TtsSpeaker(BaseModel): + name: str = Field(default="Neon", description="Name of the speaker") + speaker: Optional[str] = Field( + default=None, + description="Deprecated: Use 'name' instead", + deprecated=True) + language: str = Field(default="en-us", description="BCP-47 language code") + gender: Gender = Field(default="female", + description="Requested or synthesized gender") + voice: Optional[str] = Field( + default=None, description="Requested or synthesized voice name") + + @model_validator(mode='before') + @classmethod + def map_speaker_to_name(cls, values): + if hasattr(values, 'model_dump'): + values = values.model_dump() + if 'name' not in values and 'speaker' in values: + values['name'] = values['speaker'] + return values + + +class TtsReponseData(BaseModel): + responses: Dict[str, TtsResponse] = Field( + description="Dictionary of BCP-47 lang codes to TTS responses") + speaker: Optional[TtsSpeaker] = Field( + default=None, description="Optional speaker metadata") + + +class GetSttData(BaseModel): + audio_data: str = Field(description="Base64-encoded audio data") + lang: str = Field(default="en-us", + description="BCP-47 language code for STT") + + @model_validator(mode='before') + @classmethod + def validate_inputs(cls, values): + # Ensure input values are normalized to valid field names + if hasattr(values, 'model_dump'): + values = values.model_dump() + if 'audio_data' not in values: + values['audio_data'] = values.get('message_body') + return values + + +class SttReponseData(BaseModel): + transcripts: List[str] + parser_data: Dict[str, Any] + + @model_validator(mode='after') + def validate_transcripts_not_empty(self): + """Validate that the transcripts list is not empty.""" + if not self.transcripts: + raise ValueError("'transcripts' cannot be an empty list") + return self + + +class GetResponseData(BaseModel): + utterances: List[str] = Field(description="List of input utterance(s)") + lang: str = Field(default="en-us", + description="BCP-47 language code for input/response") + + @model_validator(mode='before') + @classmethod + def validate_inputs(cls, values): + # Ensure input values are normalized to valid field names + if hasattr(values, 'model_dump'): + values = values.model_dump() + if 'utterances' not in values: + values['utterances'] = [values.pop('messageText', '')] + return values + + +class NeonLanguagesData(BaseModel): + stt: List[str] = Field(description="List of supported STT languages") + tts: List[str] = Field(description="List of supported TTS languages") + skills: List[str] = Field(description="List of supported skill languages") + + +# Message models +class NeonGetTts(BaseMessage): + msg_type: Literal["neon.get_tts"] = "neon.get_tts" + data: GetTtsData + + @model_validator(mode='after') + def ensure_audio_destination(self): + if "audio" not in self.context.destination: + self.context.destination = ["audio"] + return self + + +class NeonGetStt(BaseMessage): + msg_type: Literal["neon.get_stt"] = "neon.get_stt" + data: GetSttData + + @model_validator(mode='after') + def ensure_audio_destination(self): + if "audio" not in self.context.destination: + # Default context is not valid for this request, so override it + self.context.destination = ["audio"] + return self + + +class NeonTextInput(BaseMessage): + msg_type: Literal["recognizer_loop:utterance"] = "recognizer_loop:utterance" + data: GetResponseData + + @model_validator(mode='after') + def ensure_skills_destination(self): + if "skills" not in self.context.destination: + self.context.destination.append("skills") + return self + + +class NeonAudioInput(BaseMessage): + msg_type: Literal["neon.audio_input"] = "neon.audio_input" + data: GetSttData + + @model_validator(mode='after') + def ensure_audio_destination(self): + if "audio" not in self.context.destination: + # Default context is not valid for this request, so override it + self.context.destination = ["audio"] + return self + + +class NeonSttResponse(BaseMessage): + msg_type: Literal["neon.get_stt.response"] = "neon.get_stt.response" + data: SttReponseData + + +class NeonTtsResponse(BaseMessage): + msg_type: Literal["neon.get_tts.response", + "klat.response"] = "neon.get_tts.response" + data: TtsReponseData + + +class NeonGetLanguages(BaseMessage): + msg_type: Literal["neon.languages.get"] = "neon.languages.get" + + +class NeonLanguagesResponse(BaseMessage): + msg_type: Literal["neon.languages.get.response"] = "neon.languages.get.response" + data: NeonLanguagesData + + +__all__ = [NeonGetTts.__name__, NeonGetStt.__name__, NeonTextInput.__name__, + NeonAudioInput.__name__, NeonSttResponse.__name__, + NeonTtsResponse.__name__, NeonGetLanguages.__name__, + NeonLanguagesResponse.__name__] diff --git a/neon_data_models/models/api/mq/__init__.py b/neon_data_models/models/api/mq/__init__.py index 9fafcf20..06db0711 100644 --- a/neon_data_models/models/api/mq/__init__.py +++ b/neon_data_models/models/api/mq/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -26,3 +26,4 @@ from neon_data_models.models.api.mq.llm import * from neon_data_models.models.api.mq.users import * +from neon_data_models.models.api.mq.neon import * diff --git a/neon_data_models/models/api/mq/llm.py b/neon_data_models/models/api/mq/llm.py index b0b97175..a40fd599 100644 --- a/neon_data_models/models/api/mq/llm.py +++ b/neon_data_models/models/api/mq/llm.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/mq/neon.py b/neon_data_models/models/api/mq/neon.py new file mode 100644 index 00000000..c2881bfc --- /dev/null +++ b/neon_data_models/models/api/mq/neon.py @@ -0,0 +1,197 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2025 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from typing import Annotated, List, Literal, Union +from pydantic import Field, TypeAdapter, model_validator + +from neon_data_models.models.base import BaseModel +from neon_data_models.models.base.contexts import KlatContext, MQContext, \ + SessionContext, TimingContext +from neon_data_models.models.base.messagebus import BaseMessage, MessageContext +from neon_data_models.models.user.database import NeonUserConfig +from neon_data_models.models.api.messagebus import NeonGetLanguages, NeonGetTts, NeonGetStt, \ + NeonAudioInput, NeonLanguagesResponse, NeonTextInput, NeonSttResponse, NeonTtsResponse + + +class GetTtsData(BaseModel): + text: str = Field(description="Text to be spoken") + lang: str = Field(default="en-us", + description="BCP-47 language code for TTS") + + @model_validator(mode='before') + @classmethod + def validate_inputs(cls, values): + if 'text' not in values: + values['text'] = values.pop('utterance', None) + return values + + +class GetSttData(BaseModel): + audio_data: str = Field(description="Base64-encoded audio data") + lang: str = Field(default="en-us", + description="BCP-47 language code for STT") + + @model_validator(mode='before') + @classmethod + def validate_inputs(cls, values): + if 'audio_data' not in values: + values['audio_data'] = values.get('message_body') + return values + + +class GetResponseData(BaseModel): + utterances: List[str] = Field(description="List of input utterance(s)") + lang: str = Field(default="en-us", + description="BCP-47 language code for input/response") + + @model_validator(mode='before') + @classmethod + def validate_inputs(cls, values): + if 'utterances' not in values: + values['utterances'] = [values.pop('messageText', '')] + return values + +class NeonMqGetTts(NeonGetTts, MQContext): + """ + Data model for an MQ message requesting TTS synthesis + """ + + +class NeonMqGetStt(NeonGetStt, MQContext): + """ + Data model for an MQ message requesting STT recognition + """ + + +class NeonMqTextInput(NeonTextInput, MQContext): + """ + Data model for an MQ message containing text input + """ + + +class NeonMqAudioInput(NeonAudioInput, MQContext): + """ + Data model for an MQ message containing audio input + """ + + +class NeonMqSttResponse(NeonSttResponse, MQContext): + """ + Data model for an MQ message containing an STT recognition response + """ + + +class NeonMqTtsResponse(NeonTtsResponse, MQContext): + """ + Data model for an MQ message containing a TTS synthesis response + """ + + +class NeonMqGetLanguages(NeonGetLanguages, MQContext): + """ + Data model for an MQ message requesting supported languages + """ + + +class NeonMqLanguagesResponse(NeonLanguagesResponse, MQContext): + """ + Data model for an MQ message containing supported languages response + """ + + +class NeonMqUnknownMessage(BaseMessage, MQContext): + """ + Default message class for validating Messagebus messages that should be + forwarded to an MQ service + """ + + +class NeonApiMessage: + """ + Type adapter for validating an arbitrary MQ message. This will always return + an instance that extends `BaseMessage` and `MQContext`. + """ + ta = TypeAdapter(Annotated[Union[NeonMqGetStt, NeonMqGetTts, + NeonMqTextInput, NeonMqSttResponse, + NeonMqTtsResponse, NeonMqGetLanguages, + NeonMqLanguagesResponse], + Field(discriminator='msg_type')]) + + @classmethod + def __new__(cls, *args, **kwargs) -> BaseMessage: + # Parse the MQ Context from a `Message` input to create a proper API message + if 'message_id' not in kwargs: + # Extract MQ context data from the message + mq_data = kwargs.get('context', {}).get('mq', {}) + # Update values with MQ context data + kwargs.update(mq_data) + + try: + return cls.ta.validate_python(kwargs) + except Exception as e: + # If validation fails, use the default message object + return NeonMqUnknownMessage(**kwargs) + + @staticmethod + def from_sio_message(sio_message: dict) -> BaseMessage: + """ + Parse a Klat SocketIO message into a valid MQ API message + """ + requested_service = sio_message.get("requested_skill", + "recognizer").lower() + if requested_service not in ["stt", "tts", "recognizer"]: + raise ValueError(f"Invalid requested service '{requested_service}'") + klat_context = KlatContext(**sio_message) + mq_context = MQContext(**sio_message) + context = MessageContext(source="mq_api", + client=sio_message.get("client", "unknown"), + username=sio_message.get("nick", "guest"), + klat_data=klat_context, mq=mq_context, + user_profiles=[NeonUserConfig()], + session=SessionContext( + session_id=sio_message.get("cid", "klat")), + timing=TimingContext( + client_sent=sio_message.get("timeCreated")) + ) + if requested_service == "stt": + context.destination = ["speech"] + return NeonMqGetStt(data=GetSttData(**sio_message), context=context, + **mq_context.model_dump()) + elif requested_service == "tts": + context.destination = ["audio"] + return NeonMqGetTts(data=GetTtsData(**sio_message), context=context, + **mq_context.model_dump()) + elif requested_service == "recognizer": + context.destination = ["skills"] + return NeonMqTextInput(data=GetResponseData(**sio_message), + context=context, **mq_context.model_dump()) + + +__all__ = [NeonMqGetTts.__name__, NeonMqGetStt.__name__, + NeonMqTextInput.__name__, NeonMqAudioInput.__name__, + NeonMqSttResponse.__name__, NeonMqTtsResponse.__name__, + NeonMqGetLanguages.__name__, NeonMqLanguagesResponse.__name__, + NeonApiMessage.__name__, NeonMqUnknownMessage.__name__] diff --git a/neon_data_models/models/api/mq/users.py b/neon_data_models/models/api/mq/users.py index 3d7b1cb7..a5b94e19 100644 --- a/neon_data_models/models/api/mq/users.py +++ b/neon_data_models/models/api/mq/users.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/node_v1/__init__.py b/neon_data_models/models/api/node_v1/__init__.py index 34405952..54700848 100644 --- a/neon_data_models/models/api/node_v1/__init__.py +++ b/neon_data_models/models/api/node_v1/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/base/__init__.py b/neon_data_models/models/base/__init__.py index 787a5b42..220c7e3b 100644 --- a/neon_data_models/models/base/__init__.py +++ b/neon_data_models/models/base/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -25,9 +25,43 @@ # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. from os import environ +from datetime import datetime, timedelta from pydantic import ConfigDict, BaseModel as _BaseModel class BaseModel(_BaseModel): model_config = ConfigDict(extra="allow" if environ.get( "NEON_DATA_MODELS_ALLOW_EXTRA", "false") != "false" else "ignore") + + + def model_dump(self, *args, **kwargs) -> dict: + """ + Global `model_dump` overrides to ensure model serialization. + Recursively processes nested dictionaries, lists, and BaseModel instances. + """ + data = super().model_dump(*args, **kwargs) + return self._process_data(data) + + def _process_data(self, data): + """ + Recursively process data to convert datetime and timedelta objects. + """ + if isinstance(data, dict): + # Process dictionary values + for key, value in list(data.items()): + data[key] = self._process_data(value) + return data + elif isinstance(data, list): + # Process list elements + return [self._process_data(item) for item in data] + elif isinstance(data, datetime): + # Convert datetime to timestamp + return data.timestamp() + elif isinstance(data, timedelta): + # Convert timedelta to seconds + return data.total_seconds() + else: + # Return other types unchanged + return data + + diff --git a/neon_data_models/models/base/contexts.py b/neon_data_models/models/base/contexts.py index f715169f..c944514e 100644 --- a/neon_data_models/models/base/contexts.py +++ b/neon_data_models/models/base/contexts.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -24,15 +24,19 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -from datetime import datetime, timedelta -from typing import Literal, List, Optional +from datetime import datetime, timedelta, timezone +from typing import Literal, List, Optional, Any, Tuple + +from pydantic import Field, model_validator from neon_data_models.models.base import BaseModel class SessionContext(BaseModel): session_id: str = "default" - active_skills: List[str] = [] + active_skills: List[Tuple[str, float]] = Field( + default=[], + description="List of tuple `skill_id` and last used timestamp") utterance_states: dict = {} lang: Optional[str] = None context: dict = {} @@ -85,13 +89,60 @@ def __init__(self, **kwargs): transform_utterance: Optional[timedelta] = None wait_in_queue: Optional[timedelta] = None + @model_validator(mode='before') + @classmethod + def validate_datetime_fields(cls, values): + """ + Validate datetime fields to parse epoch timestamps into datetime objects. + Dynamically identifies fields with datetime type annotations. + """ + # Find all fields that have datetime type annotation + datetime_fields = [ + field_name for field_name, field_info in cls.model_fields.items() + if field_info.annotation == datetime or ( + hasattr(field_info.annotation, "__origin__") and + field_info.annotation.__origin__ is Optional and + datetime in field_info.annotation.__args__ + ) + ] + + for field in datetime_fields: + if field in values and values[field] is not None: + if isinstance(values[field], (int, float)): + # Convert epoch timestamp to datetime with UTC timezone + values[field] = datetime.fromtimestamp(values[field], tz=timezone.utc) + elif not isinstance(values[field], datetime): + # If it's neither a timestamp nor a datetime, raise an error + raise ValueError(f"Field '{field}' must be a timestamp or datetime object") + + return values + class KlatContext(BaseModel): - sid: str - cid: str - title: Optional[str] = "" + sid: str = Field(default="", description="Klat Shout ID") + cid: str = Field(default="", description="Klat Conversation ID") + title: str = Field(default="", description="Klat Conversation Title") + @model_validator(mode='before') + @classmethod + def validate_inputs(cls, values): + """ + Validate KlatContext inputs to normalize messageID to sid. + """ + values.setdefault("sid", values.get("messageID", "")) + if "title" in values and values["title"] is None: + values.pop("title") + return values class MQContext(BaseModel): routing_key: Optional[str] = None message_id: str + + @model_validator(mode='before') + @classmethod + def validate_inputs(cls, values): + """ + Validate MQContext inputs to normalize messageID to message_id. + """ + values.setdefault("message_id", values.get("messageID")) + return values diff --git a/neon_data_models/models/base/messagebus.py b/neon_data_models/models/base/messagebus.py index 8773b8b6..58871b61 100644 --- a/neon_data_models/models/base/messagebus.py +++ b/neon_data_models/models/base/messagebus.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -25,7 +25,7 @@ # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. from typing import Optional, List, Union -from pydantic import ConfigDict, Field +from pydantic import ConfigDict, Field, field_validator, model_validator from neon_data_models.models.base import BaseModel from neon_data_models.models.base.contexts import (SessionContext, KlatContext, @@ -36,11 +36,12 @@ class MessageContext(BaseModel): model_config = ConfigDict(extra="allow") - session: Optional[SessionContext] = Field(description="Session Data", - default=None) + session: SessionContext = Field(description="Session Data", + default=SessionContext()) node_data: Optional[NodeData] = Field(description="Node Data", default=None) - timing: Optional[TimingContext] = Field( - description="User Interaction Timing Information", default=None) + timing: TimingContext = Field( + description="User Interaction Timing Information", + default=TimingContext()) user_profiles: Optional[List[NeonUserConfig]] = ( Field(description="List of relevant user profiles", default=None)) klat_data: Optional[KlatContext] = Field( @@ -55,11 +56,41 @@ class MessageContext(BaseModel): client_name: str = "unknown" client: str = "unknown" source: Union[str, List[str]] = "unknown" - destination: List[str] = ["skills"] + destination: Union[str, List[str]] = Field( + default=["skills"], + description="List of destination modules expected to handle the message" + ) neon_should_respond: bool = True + + @field_validator('destination') + def validate_destination(cls, v): + if isinstance(v, str): + return [v] + return v + + @model_validator(mode='before') + def validate_session_value(cls, data): + if isinstance(data, dict) and data.get('session', {}) is None: + data.pop('session') + return data class BaseMessage(BaseModel): msg_type: str data: dict context: MessageContext + + def as_messagebus_message(self) -> "Message": # type: ignore + """ + Get a `Message` representation of this object. + """ + try: + from ovos_bus_client.message import Message + if hasattr(self.data, 'model_dump'): + data = self.data.model_dump() + else: + data = self.data + return Message(self.msg_type, data, self.context.model_dump()) + except ImportError: + raise RuntimeError("pip install ovos-bus-client to enable Message " + "deserialization.") diff --git a/neon_data_models/models/client/__init__.py b/neon_data_models/models/client/__init__.py index a2d27745..e1d91a69 100644 --- a/neon_data_models/models/client/__init__.py +++ b/neon_data_models/models/client/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/client/node.py b/neon_data_models/models/client/node.py index 36873944..79880aed 100644 --- a/neon_data_models/models/client/node.py +++ b/neon_data_models/models/client/node.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/user/__init__.py b/neon_data_models/models/user/__init__.py index de6ca0de..8db545b3 100644 --- a/neon_data_models/models/user/__init__.py +++ b/neon_data_models/models/user/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/user/database.py b/neon_data_models/models/user/database.py index 071c0afe..6c92f0d5 100644 --- a/neon_data_models/models/user/database.py +++ b/neon_data_models/models/user/database.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -31,7 +31,7 @@ # from neon_data_models.models.api import HanaToken from neon_data_models.models.base import BaseModel -from pydantic import Field +from pydantic import Field, field_validator from datetime import date from neon_data_models.enum import AccessRoles @@ -49,6 +49,14 @@ class _UserConfig(BaseModel): "(i.e. `https://example.com/avatar.jpg") about: str = "" phone: str = "" + + @field_validator('dob', mode='before') + @classmethod + def validate_dob(cls, v): + if v == "YYYY/MM/DD": + # Legacy default value for dob should be treated as `None` + return None + return v class _LanguageConfig(BaseModel): @@ -197,7 +205,7 @@ def rebuild_model(cls): klat: KlatConfig = KlatConfig() llm: BrainForgeConfig = BrainForgeConfig() permissions: PermissionsConfig = PermissionsConfig() - tokens: Optional[List['HanaToken']] = [] + tokens: Optional[List['HanaToken']] = [] # type: ignore def __eq__(self, other): return self.model_dump() == other.model_dump() diff --git a/neon_data_models/models/user/neon_profile.py b/neon_data_models/models/user/neon_profile.py index c14919cf..bbb39961 100644 --- a/neon_data_models/models/user/neon_profile.py +++ b/neon_data_models/models/user/neon_profile.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/types.py b/neon_data_models/types.py new file mode 100644 index 00000000..c656b6ad --- /dev/null +++ b/neon_data_models/types.py @@ -0,0 +1,30 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2025 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from typing import Literal + + +Gender = Literal["male", "female"] diff --git a/neon_data_models/util.py b/neon_data_models/util.py index e46b10a8..fdd7aef8 100644 --- a/neon_data_models/util.py +++ b/neon_data_models/util.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 750ac949..3b16021a 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Framework # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # Contributors: Daniel McKnight, Guy Daniels, Elon Gasper, Richard Leeds, # Regina Bloomstine, Casimiro Ferreira, Andrii Pernatii, Kirill Hrymailo # BSD-3 License diff --git a/requirements/test.txt b/requirements/test.txt index 55b033e9..7b97608f 100644 --- a/requirements/test.txt +++ b/requirements/test.txt @@ -1 +1,2 @@ -pytest \ No newline at end of file +pytest +ovos-bus-client~=0.1 \ No newline at end of file diff --git a/setup.py b/setup.py index ac66e4cc..f402e395 100644 --- a/setup.py +++ b/setup.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Framework # All trademark and other rights reserved by their respective owners -# Copyright 2008-2022 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # Contributors: Daniel McKnight, Guy Daniels, Elon Gasper, Richard Leeds, # Regina Bloomstine, Casimiro Ferreira, Andrii Pernatii, Kirill Hrymailo # BSD-3 License diff --git a/tests/models/api/test_llm.py b/tests/models/api/test_llm.py index fd8a4ba1..af7e902c 100644 --- a/tests/models/api/test_llm.py +++ b/tests/models/api/test_llm.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/models/api/test_messagebus.py b/tests/models/api/test_messagebus.py new file mode 100644 index 00000000..9e91527c --- /dev/null +++ b/tests/models/api/test_messagebus.py @@ -0,0 +1,719 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2025 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from unittest import TestCase +from pydantic import ValidationError + + +class TestMessagebusModels(TestCase): + def test_get_tts_data(self): + from neon_data_models.models.api.messagebus import GetTtsData + + # Test valid data + valid_data = {"text": "Hello world", "lang": "en-us"} + tts_data = GetTtsData(**valid_data) + self.assertIsInstance(tts_data, GetTtsData) + self.assertEqual(tts_data.text, "Hello world") + self.assertEqual(tts_data.lang, "en-us") + + # Test with utterance instead of text (backward compatibility) + compat_data = {"utterance": "Hello world"} + tts_data = GetTtsData(**compat_data) + self.assertEqual(tts_data.text, "Hello world") + self.assertEqual(tts_data.lang, "en-us") # Default value + + # Test missing required fields + with self.assertRaises(ValidationError): + GetTtsData() + + def test_tts_response_data(self): + from neon_data_models.models.api.messagebus import TtsResponse, TtsReponseData + from neon_data_models.types import Gender + + # Test valid response data + valid_response = { + "sentence": "Hello world", + "translated": False, + "phonemes": "HH AH L OW W ER L D", + "genders": ["female", "male"], + "audio": {"female": "base64audio1", "male": "base64audio2"} + } + tts_response = TtsResponse(**valid_response) + self.assertIsInstance(tts_response, TtsResponse) + self.assertEqual(tts_response.sentence, "Hello world") + self.assertEqual(tts_response.translated, False) + self.assertEqual(tts_response.phonemes, "HH AH L OW W ER L D") + self.assertEqual(tts_response.genders, ["female", "male"]) + self.assertEqual(tts_response.audio["female"], "base64audio1") + self.assertEqual(tts_response.audio["male"], "base64audio2") + + # Test default values for male and female fields + self.assertIsNone(tts_response.male) + self.assertIsNone(tts_response.female) + + # Test with explicit male and female fields + valid_response_with_gender_paths = { + "sentence": "Hello world", + "translated": False, + "genders": ["female", "male"], + "audio": {"female": "base64audio1", "male": "base64audio2"}, + "male": "/path/to/male.wav", + "female": "/path/to/female.wav" + } + tts_response_with_gender_paths = TtsResponse(**valid_response_with_gender_paths) + self.assertEqual(tts_response_with_gender_paths.male, "/path/to/male.wav") + self.assertEqual(tts_response_with_gender_paths.female, "/path/to/female.wav") + + # Test with different content in audio dict vs direct fields + mixed_response = { + "sentence": "Hello world", + "translated": False, + "genders": ["female", "male"], + "audio": {"female": "base64audio1", "male": "base64audio2"}, + "male": "/path/to/different_male.wav", + "female": "/path/to/different_female.wav" + } + mixed_tts_response = TtsResponse(**mixed_response) + self.assertEqual(mixed_tts_response.audio["male"], "base64audio2") + self.assertEqual(mixed_tts_response.male, "/path/to/different_male.wav") + self.assertEqual(mixed_tts_response.audio["female"], "base64audio1") + self.assertEqual(mixed_tts_response.female, "/path/to/different_female.wav") + + # Test valid response data without phonemes + valid_response_no_phonemes = { + "sentence": "Hello world", + "translated": False, + "genders": ["female", "male"], + "audio": {"female": "base64audio1", "male": "base64audio2"} + } + tts_response_no_phonemes = TtsResponse(**valid_response_no_phonemes) + self.assertIsInstance(tts_response_no_phonemes, TtsResponse) + self.assertEqual(tts_response_no_phonemes.sentence, "Hello world") + self.assertEqual(tts_response_no_phonemes.translated, False) + self.assertIsNone(tts_response_no_phonemes.phonemes) + self.assertEqual(tts_response_no_phonemes.genders, ["female", "male"]) + self.assertEqual(tts_response_no_phonemes.audio["female"], "base64audio1") + self.assertEqual(tts_response_no_phonemes.audio["male"], "base64audio2") + + # Test valid responses data + valid_responses_data = { + "responses": { + "en-us": tts_response + } + } + tts_responses = TtsReponseData(**valid_responses_data) + self.assertIsInstance(tts_responses, TtsReponseData) + self.assertEqual(tts_responses.responses["en-us"].sentence, "Hello world") + self.assertEqual(tts_responses.responses["en-us"].genders, ["female", "male"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + TtsResponse(sentence="Hello", phonemes="HH AH L OW", translated=False) # Missing genders and audio + + with self.assertRaises(ValidationError): + TtsResponse(sentence="Hello", phonemes="HH AH L OW", translated=False, genders=["female"]) # Missing audio + + with self.assertRaises(ValidationError): + TtsReponseData() # Missing responses + + def test_get_stt_data(self): + from neon_data_models.models.api.messagebus import GetSttData + + # Test valid data + valid_data = {"audio_data": "base64encodedstring", "lang": "en-us"} + stt_data = GetSttData(**valid_data) + self.assertIsInstance(stt_data, GetSttData) + self.assertEqual(stt_data.audio_data, "base64encodedstring") + self.assertEqual(stt_data.lang, "en-us") + + # Test with message_body instead of audio_data (backward compatibility) + compat_data = {"message_body": "base64encodedstring"} + stt_data = GetSttData(**compat_data) + self.assertEqual(stt_data.audio_data, "base64encodedstring") + self.assertEqual(stt_data.lang, "en-us") # Default value + + # Test missing required fields + with self.assertRaises(ValidationError): + GetSttData() + + def test_stt_response_data(self): + from neon_data_models.models.api.messagebus import SttReponseData + + # Test valid data + valid_data = { + "transcripts": ["Hello world", "Hello word"], + "parser_data": {"confidence": 0.95} + } + stt_response_data = SttReponseData(**valid_data) + self.assertIsInstance(stt_response_data, SttReponseData) + self.assertEqual(stt_response_data.transcripts[0], "Hello world") + self.assertEqual(stt_response_data.parser_data["confidence"], 0.95) + + # Test missing required fields + with self.assertRaises(ValidationError): + SttReponseData(transcripts=["Hello world"]) # Missing parser_data + + with self.assertRaises(ValidationError): + SttReponseData(parser_data={"confidence": 0.95}) # Missing transcripts + + def test_get_response_data(self): + from neon_data_models.models.api.messagebus import GetResponseData + + # Test valid data + valid_data = {"utterances": ["How are you?"], "lang": "en-us"} + response_data = GetResponseData(**valid_data) + self.assertIsInstance(response_data, GetResponseData) + self.assertEqual(response_data.utterances, ["How are you?"]) + self.assertEqual(response_data.lang, "en-us") + + # Test with messageText instead of utterances (backward compatibility) + compat_data = {"messageText": "How are you?"} + response_data = GetResponseData(**compat_data) + self.assertEqual(response_data.utterances, ["How are you?"]) + self.assertEqual(response_data.lang, "en-us") # Default value + + # Test with empty utterances + empty_data = {"utterances": []} + response_data = GetResponseData(**empty_data) + self.assertEqual(response_data.utterances, []) + + def test_neon_get_tts(self): + from neon_data_models.models.api.messagebus import NeonGetTts, GetTtsData + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid message + message_id = "test_mid" + data = GetTtsData(text="Hello world") + valid_message = NeonGetTts(data=data, context={}) + self.assertIsInstance(valid_message, NeonGetTts) + self.assertIsInstance(valid_message, BaseMessage) + self.assertEqual(valid_message.data.text, "Hello world") + self.assertEqual(valid_message.msg_type, "neon.get_tts") + + # Test ensure_audio_destination validator + empty_dest_context = {"destination": []} + message_empty_dest = NeonGetTts(data=data, context=empty_dest_context) + self.assertIn("audio", message_empty_dest.context.destination) + + other_dest_context = {"destination": ["skills", "audio"]} + message_other_dest = NeonGetTts(data=data, context=other_dest_context) + self.assertIn("audio", message_other_dest.context.destination) + self.assertIn("skills", message_other_dest.context.destination) + + audio_dest_context = {"destination": ["audio"]} + message_audio_dest = NeonGetTts(data=data, context=audio_dest_context) + self.assertEqual(message_audio_dest.context.destination, ["audio"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + NeonGetTts(message_id=message_id, context={}) # Missing data + + def test_neon_get_stt(self): + from neon_data_models.models.api.messagebus import NeonGetStt, GetSttData + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid message + message_id = "test_mid" + data = GetSttData(audio_data="base64encodedstring") + valid_message = NeonGetStt(data=data, context={}) + self.assertIsInstance(valid_message, NeonGetStt) + self.assertIsInstance(valid_message, BaseMessage) + self.assertEqual(valid_message.data.audio_data, "base64encodedstring") + self.assertEqual(valid_message.msg_type, "neon.get_stt") + + # Test ensure_audio_destination validator + empty_dest_context = {"destination": []} + message_empty_dest = NeonGetStt(data=data, context=empty_dest_context) + self.assertIn("audio", message_empty_dest.context.destination) + + other_dest_context = {"destination": ["skills", "audio"]} + message_other_dest = NeonGetStt(data=data, context=other_dest_context) + self.assertIn("audio", message_other_dest.context.destination) + self.assertIn("skills", message_other_dest.context.destination) + + audio_dest_context = {"destination": ["audio"]} + message_audio_dest = NeonGetStt(data=data, context=audio_dest_context) + self.assertEqual(message_audio_dest.context.destination, ["audio"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + NeonGetStt(message_id=message_id, context={}) # Missing data + + def test_neon_get_response(self): + from neon_data_models.models.api.messagebus import NeonTextInput, GetResponseData + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid message + message_id = "test_mid" + data = GetResponseData(utterances=["How are you?"]) + valid_message = NeonTextInput(data=data, context={}) + self.assertIsInstance(valid_message, NeonTextInput) + self.assertIsInstance(valid_message, BaseMessage) + self.assertEqual(valid_message.data.utterances, ["How are you?"]) + self.assertEqual(valid_message.msg_type, "recognizer_loop:utterance") + + # Test ensure_skills_destination validator + empty_dest_context = {"destination": []} + message_empty_dest = NeonTextInput(data=data, context=empty_dest_context) + self.assertIn("skills", message_empty_dest.context.destination) + + other_dest_context = {"destination": ["audio"]} + message_other_dest = NeonTextInput(data=data, context=other_dest_context) + self.assertIn("skills", message_other_dest.context.destination) + self.assertIn("audio", message_other_dest.context.destination) + + skills_dest_context = {"destination": ["skills"]} + message_skills_dest = NeonTextInput(data=data, context=skills_dest_context) + self.assertEqual(message_skills_dest.context.destination, ["skills"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + NeonTextInput(message_id=message_id, context={}) # Missing data + + def test_neon_stt_response(self): + from neon_data_models.models.api.messagebus import NeonSttResponse, SttReponseData + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid message + message_id = "test_mid" + data = SttReponseData(transcripts=["Hello world"], parser_data={"confidence": 0.95}) + valid_message = NeonSttResponse(data=data, context={}) + self.assertIsInstance(valid_message, NeonSttResponse) + self.assertIsInstance(valid_message, BaseMessage) + self.assertEqual(valid_message.data.transcripts[0], "Hello world") + self.assertEqual(valid_message.msg_type, "neon.get_stt.response") + + # Test missing required fields + with self.assertRaises(ValidationError): + NeonSttResponse(message_id=message_id, context={}) # Missing data + + def test_neon_tts_response(self): + from neon_data_models.models.api.messagebus import NeonTtsResponse, TtsReponseData, TtsResponse + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid message + message_id = "test_mid" + response = TtsResponse( + sentence="Hello world", + translated=False, + phonemes="HH EH L OW", + genders=["female", "male"], + audio={"female": "base64audio1", "male": "base64audio2"} + ) + data = TtsReponseData(responses={"en-us": response}) + valid_message = NeonTtsResponse(data=data, context={}) + self.assertIsInstance(valid_message, NeonTtsResponse) + self.assertIsInstance(valid_message, BaseMessage) + self.assertEqual(valid_message.data.responses["en-us"].sentence, "Hello world") + self.assertEqual(valid_message.data.responses["en-us"].genders, ["female", "male"]) + self.assertEqual(valid_message.data.responses["en-us"].audio["female"], "base64audio1") + self.assertEqual(valid_message.msg_type, "neon.get_tts.response") + + # Test alternate msg_type + alt_message = NeonTtsResponse(data=data, message_id=message_id, context={}, + msg_type="klat.response") + self.assertEqual(alt_message.msg_type, "klat.response") + + # Test missing required fields + with self.assertRaises(ValidationError): + NeonTtsResponse(message_id=message_id, context={}) # Missing data + + def test_neon_audio_input(self): + from neon_data_models.models.api.messagebus import NeonAudioInput, GetSttData + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid message + message_id = "test_mid" + data = GetSttData(audio_data="base64encodedstring") + valid_message = NeonAudioInput(data=data, context={}) + self.assertIsInstance(valid_message, NeonAudioInput) + self.assertIsInstance(valid_message, BaseMessage) + self.assertEqual(valid_message.data.audio_data, "base64encodedstring") + self.assertEqual(valid_message.data.lang, "en-us") # Default value + self.assertEqual(valid_message.msg_type, "neon.audio_input") + + # Test ensure_audio_destination validator + empty_dest_context = {"destination": []} + message_empty_dest = NeonAudioInput(data=data, context=empty_dest_context) + self.assertIn("audio", message_empty_dest.context.destination) + + other_dest_context = {"destination": ["skills", "audio"]} + message_other_dest = NeonAudioInput(data=data, context=other_dest_context) + self.assertIn("audio", message_other_dest.context.destination) + self.assertIn("skills", message_other_dest.context.destination) + + audio_dest_context = {"destination": ["audio"]} + message_audio_dest = NeonAudioInput(data=data, context=audio_dest_context) + self.assertEqual(message_audio_dest.context.destination, ["audio"]) + + # Test with message_body instead of audio_data (backward compatibility) + compat_data = GetSttData(message_body="different_base64string") + compat_message = NeonAudioInput(data=compat_data, context={}) + self.assertEqual(compat_message.data.audio_data, "different_base64string") + + # Test missing required fields + with self.assertRaises(ValidationError): + NeonAudioInput(message_id=message_id, context={}) # Missing data + + def test_neon_languages_data(self): + from neon_data_models.models.api.messagebus import NeonLanguagesData + + # Test valid data + valid_data = { + "stt": ["en-us", "es-es", "fr-fr"], + "tts": ["en-us", "es-es"], + "skills": ["en-us"] + } + languages_data = NeonLanguagesData(**valid_data) + self.assertIsInstance(languages_data, NeonLanguagesData) + self.assertEqual(languages_data.stt, ["en-us", "es-es", "fr-fr"]) + self.assertEqual(languages_data.tts, ["en-us", "es-es"]) + self.assertEqual(languages_data.skills, ["en-us"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + NeonLanguagesData(stt=["en-us"], tts=["en-us"]) # Missing skills + + with self.assertRaises(ValidationError): + NeonLanguagesData(stt=["en-us"], skills=["en-us"]) # Missing tts + + with self.assertRaises(ValidationError): + NeonLanguagesData(tts=["en-us"], skills=["en-us"]) # Missing stt + + def test_neon_get_languages(self): + from neon_data_models.models.api.messagebus import NeonGetLanguages + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid message + valid_message = NeonGetLanguages(data={}, context={}) + self.assertIsInstance(valid_message, NeonGetLanguages) + self.assertIsInstance(valid_message, BaseMessage) + self.assertEqual(valid_message.msg_type, "neon.languages.get") + + # Test with invalid msg_type + with self.assertRaises(ValidationError): + message_with_id = NeonGetLanguages(msg_type="test_mid", + data={}, context={}) + + def test_neon_languages_response(self): + from neon_data_models.models.api.messagebus import NeonLanguagesResponse, NeonLanguagesData + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid message + languages_data = NeonLanguagesData( + stt=["en-us", "es-es"], + tts=["en-us", "fr-fr"], + skills=["en-us"] + ) + valid_message = NeonLanguagesResponse(data=languages_data, context={}) + self.assertIsInstance(valid_message, NeonLanguagesResponse) + self.assertIsInstance(valid_message, BaseMessage) + self.assertEqual(valid_message.msg_type, "neon.languages.get.response") + self.assertEqual(valid_message.data.stt, ["en-us", "es-es"]) + self.assertEqual(valid_message.data.tts, ["en-us", "fr-fr"]) + self.assertEqual(valid_message.data.skills, ["en-us"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + NeonLanguagesResponse(context={}) # Missing data + + def test_tts_speaker(self): + from neon_data_models.models.api.messagebus import TtsSpeaker + + # Test valid data + valid_data = {"name": "Neon", "language": "en-us", "gender": "female", "voice": "cmu-slt"} + speaker = TtsSpeaker(**valid_data) + self.assertIsInstance(speaker, TtsSpeaker) + self.assertEqual(speaker.name, "Neon") + self.assertEqual(speaker.language, "en-us") + self.assertEqual(speaker.gender, "female") + self.assertEqual(speaker.voice, "cmu-slt") + + # Test default values + minimal_data = {"name": "Test"} + speaker = TtsSpeaker(**minimal_data) + self.assertEqual(speaker.language, "en-us") + self.assertEqual(speaker.gender, "female") + self.assertIsNone(speaker.voice) + + # Test deprecated speaker field + deprecated_data = {"speaker": "Deprecated"} + speaker = TtsSpeaker(**deprecated_data) + self.assertEqual(speaker.name, "Deprecated") + self.assertEqual(speaker.speaker, "Deprecated") + + # Test mixed name and speaker fields (name should take precedence) + mixed_data = {"name": "Primary", "speaker": "Secondary"} + speaker = TtsSpeaker(**mixed_data) + self.assertEqual(speaker.name, "Primary") + self.assertEqual(speaker.speaker, "Secondary") + + # Test invalid gender + with self.assertRaises(ValidationError): + TtsSpeaker(name="Test", gender="invalid_gender") + + def test_get_tts_data_validation_edge_cases(self): + from neon_data_models.models.api.messagebus import GetTtsData + + # Test empty text + empty_text = {"text": ""} + tts_data = GetTtsData(**empty_text) + self.assertEqual(tts_data.text, "") + + # Test with both text and utterance (text should take precedence) + dual_fields = {"text": "Primary text", "utterance": "Secondary text"} + tts_data = GetTtsData(**dual_fields) + self.assertEqual(tts_data.text, "Primary text") + + # Test with custom language + custom_lang = {"text": "Hello", "lang": "fr-fr"} + tts_data = GetTtsData(**custom_lang) + self.assertEqual(tts_data.lang, "fr-fr") + + # Test with invalid language format (should accept but not validate format) + invalid_lang = {"text": "Hello", "lang": "invalid_lang"} + tts_data = GetTtsData(**invalid_lang) + self.assertEqual(tts_data.lang, "invalid_lang") + + def test_get_stt_data_validation_edge_cases(self): + from neon_data_models.models.api.messagebus import GetSttData + + # Test empty audio data + empty_audio = {"audio_data": ""} + stt_data = GetSttData(**empty_audio) + self.assertEqual(stt_data.audio_data, "") + + # Test with both audio_data and message_body (audio_data should take precedence) + dual_fields = {"audio_data": "Primary audio", "message_body": "Secondary audio"} + stt_data = GetSttData(**dual_fields) + self.assertEqual(stt_data.audio_data, "Primary audio") + + # Test with custom language + custom_lang = {"audio_data": "data", "lang": "de-de"} + stt_data = GetSttData(**custom_lang) + self.assertEqual(stt_data.lang, "de-de") + + def test_tts_response_edge_cases(self): + from neon_data_models.models.api.messagebus import TtsResponse + + # Test with minimum valid data + min_valid_data = { + "sentence": "Test", + "translated": True, + "genders": ["female"], + "audio": {"female": "audio_data"} + } + response = TtsResponse(**min_valid_data) + self.assertEqual(response.sentence, "Test") + self.assertTrue(response.translated) + self.assertEqual(response.genders, ["female"]) + self.assertEqual(response.audio["female"], "audio_data") + + # Test with invalid gender in genders field + with self.assertRaises(ValidationError): + TtsResponse( + sentence="Test", + translated=False, + genders=["invalid_gender"], + audio={"female": "audio_data"} + ) + + # Test with mismatch between genders and audio keys + with self.assertRaises(ValidationError): + TtsResponse( + sentence="Test", + translated=False, + genders=["male"], + audio={"female": "audio_data"} + ) + + def test_tts_response_multi_language(self): + from neon_data_models.models.api.messagebus import TtsReponseData, TtsResponse, TtsSpeaker + + # Create multiple language responses + en_response = TtsResponse( + sentence="Hello world", + translated=False, + genders=["female", "male"], + audio={"female": "en_audio_female", "male": "en_audio_male"} + ) + + es_response = TtsResponse( + sentence="Hola mundo", + translated=True, + genders=["female"], + audio={"female": "es_audio_female"} + ) + + fr_response = TtsResponse( + sentence="Bonjour le monde", + translated=True, + genders=["male"], + audio={"male": "fr_audio_male"} + ) + + # Test with multiple language responses + multi_lang_data = { + "responses": { + "en-us": en_response, + "es-es": es_response, + "fr-fr": fr_response + }, + "speaker": { + "name": "MultiLang", + "language": "en-us", + "gender": "female" + } + } + + tts_response_data = TtsReponseData(**multi_lang_data) + self.assertEqual(len(tts_response_data.responses), 3) + self.assertEqual(tts_response_data.responses["en-us"].sentence, "Hello world") + self.assertEqual(tts_response_data.responses["es-es"].sentence, "Hola mundo") + self.assertEqual(tts_response_data.responses["fr-fr"].sentence, "Bonjour le monde") + self.assertEqual(tts_response_data.speaker.name, "MultiLang") + + # Verify each language's specific properties + self.assertFalse(tts_response_data.responses["en-us"].translated) + self.assertTrue(tts_response_data.responses["es-es"].translated) + self.assertEqual(tts_response_data.responses["en-us"].genders, ["female", "male"]) + self.assertEqual(tts_response_data.responses["es-es"].genders, ["female"]) + self.assertEqual(tts_response_data.responses["fr-fr"].genders, ["male"]) + + def test_get_response_data_validation_edge_cases(self): + from neon_data_models.models.api.messagebus import GetResponseData + + # Test with a single string in messageText + text_string = {"messageText": "Single message"} + response_data = GetResponseData(**text_string) + self.assertEqual(response_data.utterances, ["Single message"]) + + # Test with both utterances and messageText (utterances should take precedence) + dual_fields = { + "utterances": ["Primary utterance"], + "messageText": "Secondary utterance" + } + response_data = GetResponseData(**dual_fields) + self.assertEqual(response_data.utterances, ["Primary utterance"]) + + # Test with multiple utterances + multi_utterance = { + "utterances": ["First utterance", "Second utterance", "Third utterance"] + } + response_data = GetResponseData(**multi_utterance) + self.assertEqual(len(response_data.utterances), 3) + self.assertEqual(response_data.utterances[1], "Second utterance") + + # Test with Unicode characters + unicode_text = {"utterances": ["こんにちは", "你好", "مرحبا"]} + response_data = GetResponseData(**unicode_text) + self.assertEqual(response_data.utterances[0], "こんにちは") + + def test_serialization_deserialization(self): + from neon_data_models.models.api.messagebus import ( + GetTtsData, NeonGetTts, TtsResponse, TtsReponseData, NeonTtsResponse + ) + import json + + # Test serialization and deserialization of GetTtsData + tts_data = GetTtsData(text="Serialize me", lang="en-us") + serialized = json.loads(tts_data.model_dump_json()) + self.assertEqual(serialized["text"], "Serialize me") + self.assertEqual(serialized["lang"], "en-us") + + deserialized = GetTtsData.model_validate(serialized) + self.assertEqual(deserialized.text, "Serialize me") + self.assertEqual(deserialized.lang, "en-us") + + # Test serialization and deserialization of complete message + response = TtsResponse( + sentence="Test response", + translated=False, + phonemes="T EH S T", + genders=["female"], + audio={"female": "test_audio_data"} + ) + + response_data = TtsReponseData(responses={"en-us": response}) + message = NeonTtsResponse(data=response_data, context={"source": "test"}) + + serialized_message = json.loads(message.model_dump_json()) + self.assertEqual(serialized_message["msg_type"], "neon.get_tts.response") + self.assertEqual(serialized_message["data"]["responses"]["en-us"]["sentence"], "Test response") + + deserialized_message = NeonTtsResponse.model_validate(serialized_message) + self.assertEqual(deserialized_message.msg_type, "neon.get_tts.response") + self.assertEqual(deserialized_message.data.responses["en-us"].sentence, "Test response") + self.assertEqual(deserialized_message.data.responses["en-us"].audio["female"], "test_audio_data") + + def test_stt_response_data_edge_cases(self): + from neon_data_models.models.api.messagebus import SttReponseData + + # Test with multiple transcripts + multi_transcript = { + "transcripts": ["First guess", "Second guess", "Third guess"], + "parser_data": {"confidence": 0.8, "source": "test_engine"} + } + stt_response = SttReponseData(**multi_transcript) + self.assertEqual(len(stt_response.transcripts), 3) + self.assertEqual(stt_response.parser_data["confidence"], 0.8) + + # Test with complex parser data + complex_parser_data = { + "transcripts": ["Hello"], + "parser_data": { + "confidence": 0.95, + "engine": "test_engine", + "metadata": { + "duration": 2.5, + "sample_rate": 16000, + "format": "wav", + "channels": 1 + }, + "alternatives": [ + {"text": "Hello", "confidence": 0.95}, + {"text": "Hell no", "confidence": 0.05} + ] + } + } + stt_response = SttReponseData(**complex_parser_data) + self.assertEqual(stt_response.parser_data["metadata"]["sample_rate"], 16000) + self.assertEqual(stt_response.parser_data["alternatives"][0]["confidence"], 0.95) + + # Test with empty transcripts list (should fail) + with self.assertRaises(ValidationError): + SttReponseData( + transcripts=[], + parser_data={"confidence": 0.9} + ) + + # Test with empty parser data (should succeed as {}is valid) + valid_empty_parser = { + "transcripts": ["Hello"], + "parser_data": {} + } + stt_response = SttReponseData(**valid_empty_parser) + self.assertEqual(len(stt_response.parser_data), 0) diff --git a/tests/models/api/test_mq.py b/tests/models/api/test_mq.py index fc977168..c4e33c6a 100644 --- a/tests/models/api/test_mq.py +++ b/tests/models/api/test_mq.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -28,7 +28,7 @@ from pydantic import ValidationError -class TestMQ(TestCase): +class TestUsersMQ(TestCase): def test_create_user_db_request(self): from neon_data_models.models.api.mq.users import UserDbRequest, CreateUserRequest @@ -126,6 +126,8 @@ def test_delete_user_db_request(self): UserDbRequest(operation="delete", username="test_user", message_id="test0") + +class TestLLMMQ(TestCase): def test_mq_llm_propose_request(self): from neon_data_models.models.api.mq.llm import LLMProposeRequest from neon_data_models.models.api.llm import LLMRequest @@ -272,5 +274,523 @@ def test_mq_llm_vote_response(self): # Invalid response data with self.assertRaises(ValidationError): LLMVoteResponse(sorted_answer_indexes=[2, 0, 1, "invalid"], - message_id=""), + message_id="") + + +class TestNeonMQ(TestCase): + def test_get_tts_data(self): + from neon_data_models.models.api.mq.neon import GetTtsData + + # Test valid data + valid_data = {"text": "Hello world", "lang": "en-us"} + tts_data = GetTtsData(**valid_data) + self.assertIsInstance(tts_data, GetTtsData) + self.assertEqual(tts_data.text, "Hello world") + self.assertEqual(tts_data.lang, "en-us") + + # Test with utterance instead of text (backward compatibility) + compat_data = {"utterance": "Hello world"} + tts_data = GetTtsData(**compat_data) + self.assertEqual(tts_data.text, "Hello world") + self.assertEqual(tts_data.lang, "en-us") # Default value + + # Test missing required fields + with self.assertRaises(ValidationError): + GetTtsData() + + def test_get_stt_data(self): + from neon_data_models.models.api.mq.neon import GetSttData + + # Test valid data + valid_data = {"audio_data": "base64encodedstring", "lang": "en-us"} + stt_data = GetSttData(**valid_data) + self.assertIsInstance(stt_data, GetSttData) + self.assertEqual(stt_data.audio_data, "base64encodedstring") + self.assertEqual(stt_data.lang, "en-us") + + # Test with message_body instead of audio_data (backward compatibility) + compat_data = {"message_body": "base64encodedstring"} + stt_data = GetSttData(**compat_data) + self.assertEqual(stt_data.audio_data, "base64encodedstring") + self.assertEqual(stt_data.lang, "en-us") # Default value + + # Test missing required fields + with self.assertRaises(ValidationError): + GetSttData() + + def test_get_response_data(self): + from neon_data_models.models.api.mq.neon import GetResponseData + + # Test valid data + valid_data = {"utterances": ["How are you?"], "lang": "en-us"} + response_data = GetResponseData(**valid_data) + self.assertIsInstance(response_data, GetResponseData) + self.assertEqual(response_data.utterances, ["How are you?"]) + self.assertEqual(response_data.lang, "en-us") + + # Test with messageText instead of utterances (backward compatibility) + compat_data = {"messageText": "How are you?"} + response_data = GetResponseData(**compat_data) + self.assertEqual(response_data.utterances, ["How are you?"]) + self.assertEqual(response_data.lang, "en-us") # Default value + + # Test with empty utterances + empty_data = {"utterances": []} + response_data = GetResponseData(**empty_data) + self.assertEqual(response_data.utterances, []) + + def test_neon_mq_get_tts(self): + from neon_data_models.models.api.mq.neon import NeonMqGetTts, GetTtsData + from neon_data_models.models.base.contexts import MQContext + + # Test valid request + message_id = "test_mid" + data = GetTtsData(text="Hello world") + valid_request = NeonMqGetTts(data=data, message_id=message_id, context={}) + self.assertIsInstance(valid_request, NeonMqGetTts) + self.assertIsInstance(valid_request, MQContext) + self.assertEqual(valid_request.data.text, "Hello world") + self.assertEqual(valid_request.message_id, message_id) + + # Test missing MQ required data + with self.assertRaises(ValidationError): + NeonMqGetTts(data=data) + + # Test missing data required + with self.assertRaises(ValidationError): + NeonMqGetTts(message_id=message_id) + + def test_neon_mq_get_stt(self): + from neon_data_models.models.api.mq.neon import NeonMqGetStt, GetSttData + from neon_data_models.models.base.contexts import MQContext + + # Test valid request + message_id = "test_mid" + data = GetSttData(audio_data="base64encodedstring") + valid_request = NeonMqGetStt(data=data, message_id=message_id, + context={}) + self.assertIsInstance(valid_request, NeonMqGetStt) + self.assertIsInstance(valid_request, MQContext) + self.assertEqual(valid_request.data.audio_data, "base64encodedstring") + self.assertEqual(valid_request.message_id, message_id) + + # Test missing MQ required data + with self.assertRaises(ValidationError): + NeonMqGetStt(data=data) + + # Test missing data required + with self.assertRaises(ValidationError): + NeonMqGetStt(message_id=message_id) + + def test_neon_mq_get_response(self): + from neon_data_models.models.api.mq.neon import NeonMqTextInput, GetResponseData + from neon_data_models.models.base.contexts import MQContext + + # Test valid request + message_id = "test_mid" + data = GetResponseData(utterances=["How are you?"]) + valid_request = NeonMqTextInput(data=data, message_id=message_id, + context={}) + self.assertIsInstance(valid_request, NeonMqTextInput) + self.assertIsInstance(valid_request, MQContext) + self.assertEqual(valid_request.data.utterances, ["How are you?"]) + self.assertEqual(valid_request.message_id, message_id) + + # Test missing MQ required data + with self.assertRaises(ValidationError): + NeonMqTextInput(data=data) + + # Test missing data required + with self.assertRaises(ValidationError): + NeonMqTextInput(message_id=message_id) + + def test_neon_mq_stt_response(self): + from neon_data_models.models.api.mq.neon import NeonMqSttResponse + # TODO: This needs defined serialized messages + # # Test valid response + # valid_response = NeonMqSttResponse( + # transcription="Hello world", + # message_id="test_mid" + # ) + # self.assertIsInstance(valid_response, NeonMqSttResponse) + # self.assertEqual(valid_response.transcription, "Hello world") + # self.assertEqual(valid_response.message_id, "test_mid") + + # # Test missing MQ required data + # with self.assertRaises(ValidationError): + # NeonMqSttResponse(transcription="Hello world") + + # # Test missing transcription data + # with self.assertRaises(ValidationError): + # NeonMqSttResponse(message_id="test_mid") + + def test_neon_mq_tts_response(self): + from neon_data_models.models.api.mq.neon import NeonMqTtsResponse + # TODO: This needs defined serialized messages + # # Test valid response + # valid_response = NeonMqTtsResponse( + # audio_data="base64encodedstring", + # message_id="test_mid" + # ) + # self.assertIsInstance(valid_response, NeonMqTtsResponse) + # self.assertEqual(valid_response.audio_data, "base64encodedstring") + # self.assertEqual(valid_response.message_id, "test_mid") + + # # Test missing MQ required data + # with self.assertRaises(ValidationError): + # NeonMqTtsResponse(audio_data="base64encodedstring") + + # # Test missing audio data + # with self.assertRaises(ValidationError): + # NeonMqTtsResponse(message_id="test_mid") + + def test_neon_api_message(self): + from neon_data_models.models.api.mq.neon import NeonApiMessage, GetTtsData, GetSttData, GetResponseData + from neon_data_models.models.api.mq.neon import NeonMqGetTts, NeonMqGetStt, NeonMqTextInput + from neon_data_models.models.api.mq.neon import NeonMqSttResponse, NeonMqTtsResponse + + # Test TTS message + tts_message = NeonApiMessage( + msg_type="neon.get_tts", + data=GetTtsData(text="Hello world"), + context={}, + message_id="test_mid" + ) + self.assertIsInstance(tts_message, NeonMqGetTts) + + # Test STT message + stt_message = NeonApiMessage( + msg_type="neon.get_stt", + data=GetSttData(audio_data="base64encodedstring"), + context={}, + message_id="test_mid" + ) + self.assertIsInstance(stt_message, NeonMqGetStt) + + # Test get response message + response_message = NeonApiMessage( + msg_type="recognizer_loop:utterance", + data=GetResponseData(utterances=["How are you?"]), + context={}, + message_id="test_mid" + ) + self.assertIsInstance(response_message, NeonMqTextInput) + + # Test STT response + stt_response = NeonApiMessage( + msg_type="neon.get_stt.response", + data={"transcripts": ["test"], + "parser_data": {}}, + context={}, + message_id="test_mid" + ) + self.assertIsInstance(stt_response, NeonMqSttResponse) + + # Test TTS response + tts_response = NeonApiMessage( + msg_type="neon.get_tts.response", + data={"responses": {"en-us": {"sentence": "test", + "translated": False, + "phonemes": "", + "genders": ["female"], + "audio": { + "female": "fake_b64_audio"}}}}, + context={}, + message_id="test_mid" + ) + self.assertIsInstance(tts_response, NeonMqTtsResponse) + + # Test from_sio_message for STT + sio_stt = { + "requested_skill": "stt", + "message_body": "base64encodedstring", + "client": "test_client", + "nick": "test_user", + "cid": "test_session", + "sid": "test_shout_id", + "timeCreated": 123456789, + "message_id": "test_mid" + } + stt_req = NeonApiMessage.from_sio_message(sio_stt) + self.assertIsInstance(stt_req, NeonMqGetStt) + self.assertEqual(stt_req.data.audio_data, "base64encodedstring") + + # Test from_sio_message for TTS + sio_tts = { + "requested_skill": "tts", + "utterance": "Hello world", + "client": "test_client", + "nick": "test_user", + "cid": "test_session", + "sid": "test_shout_id", + "timeCreated": 123456789, + "message_id": "test_mid" + } + tts_req = NeonApiMessage.from_sio_message(sio_tts) + self.assertIsInstance(tts_req, NeonMqGetTts) + self.assertEqual(tts_req.data.text, "Hello world") + + # Test from_sio_message for recognizer + sio_recognizer = { + "requested_skill": "recognizer", + "messageText": "How are you?", + "client": "test_client", + "nick": "test_user", + "cid": "test_session", + "sid": "test_shout_id", + "timeCreated": 123456789, + "message_id": "test_mid" + } + recognizer_req = NeonApiMessage.from_sio_message(sio_recognizer) + self.assertIsInstance(recognizer_req, NeonMqTextInput) + self.assertEqual(recognizer_req.data.utterances, ["How are you?"]) + + # Test invalid requested_skill + with self.assertRaises(ValueError): + NeonApiMessage.from_sio_message({"requested_skill": "invalid"}) + + def test_neon_mq_text_input(self): + from neon_data_models.models.api.mq.neon import NeonMqTextInput, GetResponseData + from neon_data_models.models.base.contexts import MQContext + + # Test valid request + message_id = "test_mid" + data = GetResponseData(utterances=["How are you?"]) + valid_request = NeonMqTextInput(data=data, message_id=message_id, + context={}) + self.assertIsInstance(valid_request, NeonMqTextInput) + self.assertIsInstance(valid_request, MQContext) + self.assertEqual(valid_request.data.utterances, ["How are you?"]) + self.assertEqual(valid_request.message_id, message_id) + + # Test missing MQ required data + with self.assertRaises(ValidationError): + NeonMqTextInput(data=data) + + # Test missing data required + with self.assertRaises(ValidationError): + NeonMqTextInput(message_id=message_id) + + def test_neon_api_message_validation(self): + from neon_data_models.models.api.mq.neon import NeonApiMessage, GetTtsData + + # Test parse_from_messagebus validator with MQ context in a nested field + message_with_mq = { + "msg_type": "neon.get_tts", + "data": {"text": "Hello world"}, + "context": {"mq": { + "message_id": "mq_mid" + }} + } + + api_message = NeonApiMessage(**message_with_mq) + self.assertEqual(api_message.message_id, "mq_mid") + + # Test with direct MQ context fields + direct_message = { + "msg_type": "neon.get_tts", + "data": {"text": "Hello world"}, + "context": {}, + "message_id": "direct_mid" + } + + direct_api_message = NeonApiMessage(**direct_message) + self.assertEqual(direct_api_message.message_id, "direct_mid") + + def test_neon_mq_audio_input(self): + from neon_data_models.models.api.mq.neon import NeonMqAudioInput, GetSttData + from neon_data_models.models.api.messagebus import NeonAudioInput + from neon_data_models.models.base.contexts import MQContext + + # Test valid request + message_id = "test_mid" + data = GetSttData(audio_data="base64encodedstring") + valid_request = NeonMqAudioInput(data=data, message_id=message_id, context={}) + self.assertIsInstance(valid_request, NeonMqAudioInput) + self.assertIsInstance(valid_request, NeonAudioInput) + self.assertIsInstance(valid_request, MQContext) + self.assertEqual(valid_request.data.audio_data, "base64encodedstring") + self.assertEqual(valid_request.message_id, message_id) + self.assertEqual(valid_request.msg_type, "neon.audio_input") + + # Test missing MQ required data + with self.assertRaises(ValidationError): + NeonMqAudioInput(data=data) + + # Test missing data required + with self.assertRaises(ValidationError): + NeonMqAudioInput(message_id=message_id) + + def test_neon_mq_get_languages(self): + from neon_data_models.models.api.mq.neon import NeonMqGetLanguages + from neon_data_models.models.api.messagebus import NeonGetLanguages + from neon_data_models.models.base.contexts import MQContext + + # Test valid request + message_id = "test_mid" + valid_request = NeonMqGetLanguages(message_id=message_id, data={}, + context={}) + self.assertIsInstance(valid_request, NeonMqGetLanguages) + self.assertIsInstance(valid_request, NeonGetLanguages) + self.assertIsInstance(valid_request, MQContext) + self.assertEqual(valid_request.message_id, message_id) + self.assertEqual(valid_request.msg_type, "neon.languages.get") + + # Test missing MQ required data + with self.assertRaises(ValidationError): + NeonMqGetLanguages() + + def test_neon_mq_languages_response(self): + from neon_data_models.models.api.mq.neon import NeonMqLanguagesResponse + from neon_data_models.models.api.messagebus import NeonLanguagesResponse, NeonLanguagesData + from neon_data_models.models.base.contexts import MQContext + + # Test valid response + message_id = "test_mid" + data = NeonLanguagesData( + stt=["en-us", "es-es"], + tts=["en-us", "fr-fr"], + skills=["en-us"] + ) + valid_response = NeonMqLanguagesResponse(data=data, message_id=message_id, context={}) + self.assertIsInstance(valid_response, NeonMqLanguagesResponse) + self.assertIsInstance(valid_response, NeonLanguagesResponse) + self.assertIsInstance(valid_response, MQContext) + self.assertEqual(valid_response.message_id, message_id) + self.assertEqual(valid_response.msg_type, "neon.languages.get.response") + self.assertEqual(valid_response.data.stt, ["en-us", "es-es"]) + self.assertEqual(valid_response.data.tts, ["en-us", "fr-fr"]) + self.assertEqual(valid_response.data.skills, ["en-us"]) + + # Test missing MQ required data + with self.assertRaises(ValidationError): + NeonMqLanguagesResponse(data=data) + + # Test missing data required + with self.assertRaises(ValidationError): + NeonMqLanguagesResponse(message_id=message_id) + + def test_neon_api_message_languages(self): + from neon_data_models.models.api.mq.neon import NeonApiMessage + from neon_data_models.models.api.mq.neon import NeonMqGetLanguages, NeonMqLanguagesResponse + from neon_data_models.models.api.messagebus import NeonLanguagesData + + # Test GetLanguages message + languages_get_message = NeonApiMessage( + msg_type="neon.languages.get", + data={}, + context={}, + message_id="test_mid" + ) + self.assertIsInstance(languages_get_message, NeonMqGetLanguages) + self.assertEqual(languages_get_message.message_id, "test_mid") + self.assertEqual(languages_get_message.msg_type, "neon.languages.get") + + # Test LanguagesResponse message + languages_data = NeonLanguagesData( + stt=["en-us", "es-es"], + tts=["en-us", "fr-fr"], + skills=["en-us"] + ) + languages_response = NeonApiMessage( + msg_type="neon.languages.get.response", + data=languages_data, + context={}, + message_id="test_mid" + ) + self.assertIsInstance(languages_response, NeonMqLanguagesResponse) + self.assertEqual(languages_response.data.stt, ["en-us", "es-es"]) + self.assertEqual(languages_response.data.tts, ["en-us", "fr-fr"]) + self.assertEqual(languages_response.data.skills, ["en-us"]) + + def test_neon_mq_unknown_message(self): + from neon_data_models.models.api.mq.neon import NeonMqUnknownMessage + from neon_data_models.models.base.messagebus import BaseMessage + from neon_data_models.models.base.contexts import MQContext + + # Test valid initialization + unknown_message = NeonMqUnknownMessage( + msg_type="unknown.type", + message_id="test_mid", + data={}, + context={} + ) + + # Test inheritance + self.assertIsInstance(unknown_message, NeonMqUnknownMessage) + self.assertIsInstance(unknown_message, BaseMessage) + self.assertIsInstance(unknown_message, MQContext) + + # Test properties + self.assertEqual(unknown_message.msg_type, "unknown.type") + self.assertEqual(unknown_message.message_id, "test_mid") + + # Test with missing required fields + with self.assertRaises(ValidationError): + NeonMqUnknownMessage() + + def test_neon_api_message_fallback(self): + from neon_data_models.models.api.mq.neon import NeonApiMessage, NeonMqUnknownMessage + + # Test with unknown message type + unknown_message_data = { + "msg_type": "unknown.message.type", + "message_id": "test_mid", + "data": {"some_field": "some_value"}, + "context": {} + } + + message = NeonApiMessage(**unknown_message_data) + + # Verify fallback to NeonMqUnknownMessage + self.assertIsInstance(message, NeonMqUnknownMessage) + self.assertEqual(message.msg_type, "unknown.message.type") + self.assertEqual(message.message_id, "test_mid") + + # Test with the specific problematic message type from the error + ovos_stt_message = { + "msg_type": "ovos.languages.stt.response", + "message_id": "test_mid", + "data": {"languages": ["en-us", "es-es"]}, + "context": {} + } + + message = NeonApiMessage(**ovos_stt_message) + + # Verify it doesn't raise an exception and preserves the data + self.assertIsInstance(message, NeonMqUnknownMessage) + self.assertEqual(message.msg_type, "ovos.languages.stt.response") + self.assertEqual(message.message_id, "test_mid") + self.assertEqual(message.data, {"languages": ["en-us", "es-es"]}) + + def test_neon_api_message_with_nested_data(self): + from neon_data_models.models.api.mq.neon import NeonApiMessage, NeonMqUnknownMessage + + # Test with deeply nested data + complex_message = { + "msg_type": "complex.unknown.type", + "message_id": "test_mid", + "data": { + "nested": { + "deeply": { + "value": "test" + } + }, + "list_data": [1, 2, 3, {"key": "value"}] + }, + "context": { + "client": "test_client", + "user": "test_user" + } + } + + message = NeonApiMessage(**complex_message) + + # Verify complex data is preserved + self.assertIsInstance(message, NeonMqUnknownMessage) + self.assertEqual(message.msg_type, "complex.unknown.type") + self.assertEqual(message.message_id, "test_mid") + self.assertEqual(message.data["nested"]["deeply"]["value"], "test") + self.assertEqual(message.data["list_data"][3]["key"], "value") + self.assertEqual(message.context.client, "test_client") diff --git a/tests/models/api/test_node.py b/tests/models/api/test_node.py index 0039e398..255be521 100644 --- a/tests/models/api/test_node.py +++ b/tests/models/api/test_node.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/models/test_base.py b/tests/models/test_base.py index 0665af88..7d5bd337 100644 --- a/tests/models/test_base.py +++ b/tests/models/test_base.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -26,7 +26,9 @@ import importlib import os -from datetime import datetime, timedelta +import json + +from datetime import datetime, timedelta, timezone from unittest import TestCase from time import time from pydantic import ValidationError @@ -101,36 +103,265 @@ def test_session_context(self): def test_timing_context(self): from neon_data_models.models.base.contexts import TimingContext + default = TimingContext() self.assertIsNone(default.model_dump()['handle_utterance']) # Alias handling test_time = time() - test_duration = 0.00001234 timing = TimingContext(transcribed=test_time, - text_parsers=test_duration) + text_parsers=0.0001) # Type casting self.assertIsInstance(timing.handle_utterance, datetime) - self.assertAlmostEqual(timing.handle_utterance.timestamp(), test_time, 5) + self.assertAlmostEqual(timing.handle_utterance.timestamp(), + test_time, 0) self.assertIsInstance(timing.transform_utterance, timedelta) - self.assertAlmostEqual(timing.transform_utterance.total_seconds(), test_duration, 5) + self.assertAlmostEqual(timing.transform_utterance.total_seconds(), + 0, 0) # Dump/Load serialized = timing.model_dump() self.assertEqual(serialized['handle_utterance'], - timing.handle_utterance) + timing.handle_utterance.timestamp()) self.assertEqual(serialized['transform_utterance'], - timing.transform_utterance) + timing.transform_utterance.total_seconds()) self.assertEqual(timing, TimingContext(**serialized)) + # Create a TimingContext with sample data + now = datetime.now(tz=timezone.utc) + one_second = timedelta(seconds=1) + + context = TimingContext( + audio_begin=now, + audio_end=now + one_second, + get_stt=one_second, + get_tts=timedelta(seconds=2) + ) + + # Test serialization + serialized = context.model_dump() + + # Check that datetime fields are converted to timestamps + self.assertIsInstance(serialized["audio_begin"], float) + self.assertIsInstance(serialized["audio_end"], float) + self.assertAlmostEqual(serialized["audio_begin"], + now.timestamp(), delta=0.01) + self.assertAlmostEqual(serialized["audio_end"], + (now + one_second).timestamp(), delta=0.01) + + # Check that timedelta fields are converted to seconds + self.assertIsInstance(serialized["get_stt"], float) + self.assertIsInstance(serialized["get_tts"], float) + self.assertEqual(serialized["get_stt"], 1.0) + self.assertEqual(serialized["get_tts"], 2.0) + + # Test JSON serialization + json_str = json.dumps(serialized) + self.assertTrue(json_str) # Ensure it can be JSON serialized + + # Test deserialization + deserialized_dict = json.loads(json_str) + deserialized = TimingContext(**deserialized_dict) + + # Check that timestamps are converted back to datetime objects + self.assertIsInstance(deserialized.audio_begin, datetime) + self.assertIsInstance(deserialized.audio_end, datetime) + self.assertAlmostEqual((deserialized.audio_begin - + now).total_seconds(), 0, delta=0.01) + self.assertAlmostEqual((deserialized.audio_end - + (now + one_second)).total_seconds(), 0, + delta=0.01) + + # Check that second values are converted back to timedelta objects + self.assertIsInstance(deserialized.get_stt, timedelta) + self.assertIsInstance(deserialized.get_tts, timedelta) + self.assertEqual(deserialized.get_stt.total_seconds(), 1.0) + self.assertEqual(deserialized.get_tts.total_seconds(), 2.0) + + + # Test invalid type for datetime field (string) + with self.assertRaises(ValidationError) as context: + TimingContext(audio_begin="not a timestamp or datetime") + self.assertIn("audio_begin", str(context.exception)) + + # Test invalid type for datetime field (dict) + with self.assertRaises(ValidationError) as context: + TimingContext(client_sent={"invalid": "value"}) + self.assertIn("client_sent", str(context.exception)) + + # Test invalid type for timedelta field + with self.assertRaises(ValidationError) as context: + TimingContext(get_stt="invalid") + + # Create a context with all fields populated + now = datetime.now(tz=timezone.utc) + delta = timedelta(seconds=0.5) + + timing = TimingContext( + # Datetime fields + audio_begin=now, + audio_end=now + delta, + client_sent=now + delta * 2, + gradio_sent=now + delta * 3, + handle_utterance=now + delta * 4, + response_sent=now + delta * 5, + speech_start=now + delta * 6, + + # Timedelta fields + get_stt=delta, + get_tts=delta * 2, + iris_input_handling=delta * 3, + mq_response_handler=delta * 4, + mq_from_core=delta * 5, + mq_from_client=delta * 6, + mq_input_handler=delta * 7, + client_to_core=delta * 8, + client_from_core=delta * 9, + save_transcript=delta * 10, + transform_audio=delta * 11, + transform_utterance=delta * 12, + wait_in_queue=delta * 13 + ) + + # Verify all fields are properly typed + self.assertIsInstance(timing.audio_begin, datetime) + self.assertIsInstance(timing.audio_end, datetime) + self.assertIsInstance(timing.client_sent, datetime) + self.assertIsInstance(timing.gradio_sent, datetime) + self.assertIsInstance(timing.handle_utterance, datetime) + self.assertIsInstance(timing.response_sent, datetime) + self.assertIsInstance(timing.speech_start, datetime) + + self.assertIsInstance(timing.get_stt, timedelta) + self.assertIsInstance(timing.get_tts, timedelta) + self.assertIsInstance(timing.iris_input_handling, timedelta) + self.assertIsInstance(timing.mq_response_handler, timedelta) + self.assertIsInstance(timing.mq_from_core, timedelta) + self.assertIsInstance(timing.mq_from_client, timedelta) + self.assertIsInstance(timing.mq_input_handler, timedelta) + self.assertIsInstance(timing.client_to_core, timedelta) + self.assertIsInstance(timing.client_from_core, timedelta) + self.assertIsInstance(timing.save_transcript, timedelta) + self.assertIsInstance(timing.transform_audio, timedelta) + self.assertIsInstance(timing.transform_utterance, timedelta) + self.assertIsInstance(timing.wait_in_queue, timedelta) + + # Test serialization/deserialization with all fields + serialized = timing.model_dump() + deserialized = TimingContext(**serialized) + self.assertEqual(timing, deserialized) + + + # Zero timestamp + zero_time = TimingContext(audio_begin=0) + self.assertEqual(zero_time.audio_begin, + datetime.fromtimestamp(0, tz=timezone.utc)) + + # Negative timestamp (represents dates before 1970) + neg_time = TimingContext(audio_begin=-86400) # One day before epoch + self.assertEqual(neg_time.audio_begin, + datetime.fromtimestamp(-86400, tz=timezone.utc)) + + # Very large timestamp + future_time = TimingContext(audio_begin=2147483647) # Year 2038 + self.assertEqual(future_time.audio_begin, + datetime.fromtimestamp(2147483647, tz=timezone.utc)) + + # Zero timedelta + zero_delta = TimingContext(get_stt=0) + self.assertEqual(zero_delta.get_stt, timedelta(0)) + + # Negative timedelta + neg_delta = TimingContext(get_stt=-1.5) + self.assertEqual(neg_delta.get_stt, timedelta(seconds=-1.5)) + + + now = datetime.now(tz=timezone.utc) + timestamp = now.timestamp() + + # Test with timestamp input + timestamp_input = TimingContext(audio_begin=timestamp) + self.assertIsInstance(timestamp_input.audio_begin, datetime) + self.assertAlmostEqual(timestamp_input.audio_begin.timestamp(), + timestamp, places=3) + + # Test with datetime input + datetime_input = TimingContext(audio_begin=now) + self.assertIsInstance(datetime_input.audio_begin, datetime) + self.assertAlmostEqual(datetime_input.audio_begin.timestamp(), + timestamp, places=3) + + # Compare both inputs + self.assertAlmostEqual(timestamp_input.audio_begin.timestamp(), + datetime_input.audio_begin.timestamp(), + places=3) + + # Test with timedelta object vs number of seconds + delta = timedelta(seconds=1.5) + delta_input = TimingContext(get_stt=delta) + seconds_input = TimingContext(get_stt=1.5) + + self.assertEqual(delta_input.get_stt, delta) + self.assertEqual(seconds_input.get_stt, delta) + + + timestamp = datetime.now(tz=timezone.utc).timestamp() + + # Test transcribed alias for handle_utterance + with_transcribed = TimingContext(transcribed=timestamp) + self.assertIsInstance(with_transcribed.handle_utterance, datetime) + self.assertAlmostEqual(with_transcribed.handle_utterance.timestamp(), + timestamp, places=3) + + # Test text_parsers alias for transform_utterance + with_text_parsers = TimingContext(text_parsers=1.5) + self.assertIsInstance(with_text_parsers.transform_utterance, timedelta) + self.assertEqual(with_text_parsers.transform_utterance, + timedelta(seconds=1.5)) + + # Test both aliases together + with_both = TimingContext(transcribed=timestamp, text_parsers=2.5) + self.assertIsInstance(with_both.handle_utterance, datetime) + self.assertIsInstance(with_both.transform_utterance, timedelta) + self.assertAlmostEqual(with_both.handle_utterance.timestamp(), + timestamp, places=3) + self.assertEqual(with_both.transform_utterance, + timedelta(seconds=2.5)) + + # Ensure serialized data doesn't contain the old field names + serialized = with_both.model_dump() + self.assertIn("handle_utterance", serialized) + self.assertIn("transform_utterance", serialized) + self.assertNotIn("transcribed", serialized) + self.assertNotIn("text_parsers", serialized) + def test_klat_context(self): from neon_data_models.models.base.contexts import KlatContext with self.assertRaises(ValidationError): - KlatContext() + KlatContext(sid=None) minimal_ctx = KlatContext(cid="conversation", sid="shout") self.assertEqual(minimal_ctx, KlatContext(**minimal_ctx.model_dump())) + + # Test messageID normalization to sid + message_id_ctx = KlatContext(cid="conversation", messageID="message_id_value") + self.assertEqual(message_id_ctx.sid, "message_id_value") + self.assertEqual(message_id_ctx.cid, "conversation") + + # Test serialization with normalized field + serialized = message_id_ctx.model_dump() + self.assertEqual(serialized["sid"], "message_id_value") + self.assertNotIn("messageID", serialized) + + # Test round-trip serialization + deserialized = KlatContext(**serialized) + self.assertEqual(deserialized.sid, "message_id_value") + self.assertEqual(deserialized, message_id_ctx) + + # Test that sid takes precedence when both fields are provided + both_fields_ctx = KlatContext(cid="conversation", sid="shout_id", messageID="message_id_value") + self.assertEqual(both_fields_ctx.sid, "shout_id") def test_mq_context(self): from neon_data_models.models.base.contexts import MQContext @@ -143,6 +374,7 @@ def test_mq_context(self): class TestMessagebus(TestCase): def test_base_model(self): + from ovos_bus_client.message import Message from neon_data_models.models.base.messagebus import BaseMessage from neon_data_models.models.client import NodeData from neon_data_models.models.user import NeonUserConfig @@ -164,6 +396,12 @@ def test_base_model(self): self.assertIsInstance(message.context.node_data, NodeData) self.assertIsInstance(message.context.user_profiles[0], NeonUserConfig) + as_messagebus = message.as_messagebus_message() + self.assertIsInstance(as_messagebus, Message) + self.assertEqual(as_messagebus.msg_type, message.msg_type) + self.assertEqual(as_messagebus.data, message.data) + self.assertEqual(as_messagebus.context, message.context.model_dump()) + serialized = message.model_dump() # Extra context keys are always retained for compat. self.assertEqual(serialized["context"]["extra_key"], "text") @@ -205,3 +443,49 @@ def test_message_context(self): # Round-trip serialization results in the same object self.assertEqual(extra_context, MessageContext(**serialized)) + + # Test destination validation + # Test default initialization + default_context = MessageContext() + self.assertEqual(default_context.destination, ["skills"]) + + # Test string input gets converted to list + string_context = MessageContext(destination="test") + self.assertEqual(string_context.destination, ["test"]) + + # Test list input remains a list + list_context = MessageContext(destination=["test1", "test2"]) + self.assertEqual(list_context.destination, ["test1", "test2"]) + + # Test serialization/deserialization + serialized = string_context.model_dump() + self.assertEqual(serialized["destination"], ["test"]) + + # Test round-trip + deserialized = MessageContext(**serialized) + self.assertEqual(deserialized.destination, ["test"]) + + # Test session=None validation + none_session_context = MessageContext(session=None) + self.assertIsNotNone(none_session_context.session) + self.assertIsInstance(none_session_context.session, SessionContext) + + # Test that the default is used when None is provided + default_session = SessionContext() + self.assertEqual(none_session_context.session.model_dump(), + default_session.model_dump()) + + # Test serialization with None session that gets defaulted + serialized = none_session_context.model_dump(exclude_none=True) + self.assertIn("session", serialized) + self.assertEqual(serialized["session"], default_session.model_dump()) + + # Test round-trip serialization + deserialized = MessageContext(**serialized) + self.assertEqual(deserialized.session.model_dump(), + default_session.model_dump()) + + # Test when None is passed in a nested dictionary + nested_none_context = MessageContext(**{"session": None}) + self.assertIsNotNone(nested_none_context.session) + self.assertIsInstance(nested_none_context.session, SessionContext) diff --git a/tests/models/test_client.py b/tests/models/test_client.py index 79f0a873..ca7fc5c6 100644 --- a/tests/models/test_client.py +++ b/tests/models/test_client.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/models/test_user.py b/tests/models/test_user.py index c857d178..d03b46ab 100644 --- a/tests/models/test_user.py +++ b/tests/models/test_user.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -45,6 +45,9 @@ def test_neon_user_config(self): "preferred_name": "tester", "dob": "2001-01-01", "email": "developers@neon.ai", + "avatar_url": "https://example.com/avatar.jpg", + "about": "Test user biography", + "phone": "555-123-4567", "extra_key": "This should be removed by validation" }, language={"input": ["en-us", "uk-ua"], @@ -67,6 +70,17 @@ def test_neon_user_config(self): extras_keys = getattr(valid_with_extras, section).model_dump().keys() self.assertEqual(default_keys, extras_keys) + # Testing user fields specifically + self.assertEqual(valid_with_extras.user.first_name, "Daniel") + self.assertEqual(valid_with_extras.user.middle_name, "James") + self.assertEqual(valid_with_extras.user.last_name, "McKnight") + self.assertEqual(valid_with_extras.user.preferred_name, "tester") + self.assertEqual(valid_with_extras.user.email, "developers@neon.ai") + self.assertEqual(valid_with_extras.user.avatar_url, "https://example.com/avatar.jpg") + self.assertEqual(valid_with_extras.user.about, "Test user biography") + self.assertEqual(valid_with_extras.user.phone, "555-123-4567") + self.assertEqual(valid_with_extras.user.dob, date(2001, 1, 1)) + # Validation errors with self.assertRaises(ValidationError): NeonUserConfig(units={"time": 13}) @@ -84,6 +98,52 @@ def test_neon_user_config(self): config = NeonUserConfig(user={"dob": "2001-01-01"}) self.assertIsInstance(config.user.dob, date) + def test_user_config(self): + """Test the _UserConfig class specifically.""" + from neon_data_models.models.user.database import _UserConfig + + # Test default values + default_config = _UserConfig() + self.assertEqual(default_config.first_name, "") + self.assertEqual(default_config.middle_name, "") + self.assertEqual(default_config.last_name, "") + self.assertEqual(default_config.preferred_name, "") + self.assertEqual(default_config.email, "") + self.assertEqual(default_config.avatar_url, "") + self.assertEqual(default_config.about, "") + self.assertEqual(default_config.phone, "") + self.assertIsNone(default_config.dob) + + # Test setting values + user_config = _UserConfig( + first_name="John", + middle_name="Doe", + last_name="Smith", + preferred_name="Johnny", + email="john.smith@example.com", + avatar_url="https://example.com/avatar.jpg", + about="Test biography", + phone="555-987-6543", + dob=date(1990, 1, 1) + ) + self.assertEqual(user_config.first_name, "John") + self.assertEqual(user_config.middle_name, "Doe") + self.assertEqual(user_config.last_name, "Smith") + self.assertEqual(user_config.preferred_name, "Johnny") + self.assertEqual(user_config.email, "john.smith@example.com") + self.assertEqual(user_config.avatar_url, "https://example.com/avatar.jpg") + self.assertEqual(user_config.about, "Test biography") + self.assertEqual(user_config.phone, "555-987-6543") + self.assertEqual(user_config.dob, date(1990, 1, 1)) + + # Test dob validator + legacy_config = _UserConfig(dob="YYYY/MM/DD") + self.assertIsNone(legacy_config.dob) + + # Test valid date string + valid_date_config = _UserConfig(dob="2000-12-31") + self.assertEqual(valid_date_config.dob, date(2000, 12, 31)) + def test_user(self): from neon_data_models.enum import AccessRoles user_kwargs = dict(username="test", diff --git a/tests/test_enum.py b/tests/test_enum.py index c95b6d37..a3f64329 100644 --- a/tests/test_enum.py +++ b/tests/test_enum.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/test_imports.py b/tests/test_imports.py index fcf3c0d5..4fa67cdc 100644 --- a/tests/test_imports.py +++ b/tests/test_imports.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2024 Neongecko.com Inc. +# Copyright 2008-2025 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: From ed70fdfd1d3191461a71db3733d3c64014d35387 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Thu, 27 Mar 2025 19:50:09 +0000 Subject: [PATCH 02/36] Increment Version to 0.0.2a1 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 3b16021a..47f0f122 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.1" +__version__ = "0.0.2a1" From 26281cff04afc0cce034acf09c1406dae9bfe8c5 Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Fri, 4 Apr 2025 10:03:41 -0700 Subject: [PATCH 03/36] Implement MQ models for chatbots API (#19) * Define `ChatbotRequest` model with Claude-written unit tests * Use `userID` when `userDisplayName` is not specified in SIO message with unit test coverage * Add `messageText` to serialized `ChatbotRequest` for backwards-compat. * Add more backwards-compat. data to serialized `ChatbotRequest` objects * Implement `ChatbotResponse` model with unit test coverage * Add models for `ChatbotSavePrompt` and `ChatbotNewPrompt` * Update `ChatbotNewPrompt` based on test failure * Resolve `user_id` field override * Define `PromptCompletedContext` and update `ChatbotSavePrompt` model based on Klat server logged error * Complete `PromptCompletedContext` based on observed data structure * Troubleshooting server message handling * Refactor model names to include `Mq` for consistency Update unit tests to cover added models in `chatbots` module Update Chatbots MQ models to allow field names in addition to aliases Update Chatbots MQ models to include both field names and aliases in serialized outputs * Implements `SubmindStatus` and `CcaiState` enum classes from other modules Implements `BotType` literal type annotation Adds `ChatbotsMqSubmindsState` model with unit tests * Add validator to resolve observed issues with `ConnectedSubmind` validation * Add global handling to serialize `Enum` objects by value * Update `LLMPersonaIdentity` model to validate API inputs that use `persona_name` as a key Define models for configured personas requests with unit test coverage * Add `user_id` param to personas request * Update PersonasResponse.from_persona_request to use request object to copy MQ context with unit test updates * Add literal `msg_type` field for SIO compat * Implement Chatbots prompts API data moels with unit tests * Enable default `message_id` value for MQContext with updated unit test * Fix default `message_id` handling * Fix default factory for message_id * Alias both `userID` and `nick` to `user_id` for backwards-compat. * Update `ChatbotsMqResponse` based on observed ValidationErrors from chatbots-observer * Troubleshooting response aliases * Annotate deprecated params to resolve errors * Fix logical error in `ChatbotsMqResponse` validator method * Troubleshoot backwards-compat. * More model validation troubleshooting * Prevent validating inputs that aren't `dict` objects * Fix `is_announcement` type annotation in `ChatbotsMqResponse` based on observed value * Update `isAnnouncement` serialization to use `'0'` and `'1'` for backwards-compat. * Troubleshoot validation failure * Troubleshooting `sid` handling * Troubleshooting `BotType` value handling * Alias `is_bot` for `ChatbotsMqResponse` to troubleshoot response handling * Add params to `ChatbotsMqResponse` to troubleshoot missing params * Fix serialization patch * Add `created_on` and `to_discussion` to `ChatbotMqResponse` to troubleshoot missing responses * Alias `prompt_state` to `conversation_state` based on observed chatbots-obsersver behavior * Troubleshooting missing context * Add validator to `ChatbotsMqSavePrompt` to address errors observed in testing * Add models for Submind connection/disconnection * Refactor `nick` to `user_id` for consistency with other models * Implement submind invitation events * Add models for submind ban events * Implement `CcaiControl` enum and `ChatbotsMqResponse` discriminator model Add test coverage by Claude 3.7 * Troubleshoot `CcaiControl` enum references * Update tests to reflect current implementation * Troubleshooting `ChatbotsMqResponse` based on testing Update unit tests to use actual example data * Fix failing unit test * Troubleshooting errors with message handling in `pyklatchat-observer` module * Dump `prompt_state` to `promptState` for SIO compat. * Update `ChatbotsMqResponse.__new__` to resolve observed issues in chatbots observer Update unit tests with more real data test cases * Update `ChatbotsMqNewPrompt` input value handling with unit test * Implement Error model * Cleanup `chatbots` module to remove unused parameters with unit test updates * Refactor `populate_by_name` config to `BaseModel` with envvar handling Update `created_on` param handling to reduce duplicated params * Continue annotating new context params Remove unused params from data models Simplify model validation Fix indent in unit test sample data for readability * Finish annotating fields in `chatbots` module Resolve unit test failure (leftover from before last refactor) Clean up to prep for review/release * Implement default values for backwards-compat. * Update `prompt_state` annotation to CcaiState instead of `int` * Add back deprecated `context` field for Klat Server compat. * Troubleshooting Klat server compat. * Annotate `bot_type` as optional so default value is valid Update `prompt_state` alias handling for backwards-compat. * Override `prompt_state` for `ChatbotsMqNewPrompt` to address observed validation failure with current chatbots-observer module * Update unit test to account for behavior change * Add docstrings to added `enum` classes Refactor LLMPersonaIdentity validator to an alias with unit test coverage * Cleanup alias handling Update Field annotations Reorganize deprecated fields Remove commented blocks Address remaining TODO notes * Add missing `,` * Remove invalid mocked data from test cases * Fix backwards-compat `model_dump` in `ChatbotsMqRequest` * Refactor `ConnectedSubmind` and `ChatbotsMqSubmindConnection` for `mq-chatbots-observer` updates * Implement default value for `ConnectedSubmind.attached_cids` for backwards-compat. * Update tests to address failure * Troubleshooting default value handling * Fix manual serialized value * Fix error in default `last_ping` param * Refactor `ConnectedSubmind` into its own module with Claude-generated unit test case * Fix indent error Add test coverage for `ConnectedSubmind` context handling * Make `service_name` context determination safe * Fix typo in comment --- neon_data_models/enum.py | 33 +- neon_data_models/models/api/chatbots.py | 55 + neon_data_models/models/api/llm.py | 3 +- neon_data_models/models/api/mq/__init__.py | 3 +- neon_data_models/models/api/mq/chatbots.py | 446 +++++++ neon_data_models/models/base/__init__.py | 9 +- neon_data_models/models/base/contexts.py | 7 +- neon_data_models/types.py | 2 + tests/models/api/test_chatbots.py | 73 ++ tests/models/api/test_llm.py | 5 + tests/models/api/test_mq.py | 1347 +++++++++++++++++++- tests/models/test_base.py | 5 +- 12 files changed, 1962 insertions(+), 26 deletions(-) create mode 100644 neon_data_models/models/api/chatbots.py create mode 100644 neon_data_models/models/api/mq/chatbots.py create mode 100644 tests/models/api/test_chatbots.py diff --git a/neon_data_models/enum.py b/neon_data_models/enum.py index d228f1ae..eface825 100644 --- a/neon_data_models/enum.py +++ b/neon_data_models/enum.py @@ -24,7 +24,7 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -from enum import IntEnum +from enum import Enum, IntEnum class AccessRoles(IntEnum): @@ -92,3 +92,34 @@ class Weekdays(IntEnum): FRI = 4 SAT = 5 SUN = 6 + + +class SubmindStatus(Enum): + """ + Defines status of a submind in a particular conversation + """ + BANNED = "banned" + ACTIVE = "active" + + +class CcaiState(IntEnum): + """ + Defines status of a conversation where a CCAI is operating + """ + IDLE = 0 # No active prompt + RESP = 1 # Gathering responses to prompt + DISC = 2 # Discussing responses + VOTE = 3 # Voting on responses + PICK = 4 # Proctor will select response + WAIT = 5 # Bot is waiting for an invitation to participate + + +class CcaiControl(Enum): + """ + Defines control messages for a CCAI conversation + """ + PROMPT_REGEX = r"^!PROMPT:" + SAVE_PROMPT_RESULTS = "!MSG:SAVE_PROMPT_RESULTS" + CREATE_PROMPT = "!MSG:CREATE_PROMPT" + START_AUTO_PROMPTS = "!START_AUTO_PROMPTS" + STOP_AUTO_PROMPTS = "!STOP_AUTO_PROMPTS" diff --git a/neon_data_models/models/api/chatbots.py b/neon_data_models/models/api/chatbots.py new file mode 100644 index 00000000..4f31bbb7 --- /dev/null +++ b/neon_data_models/models/api/chatbots.py @@ -0,0 +1,55 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2025 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from typing import List, Optional +from datetime import datetime, timezone +from pydantic import Field +from neon_data_models.models.base import BaseModel +from neon_data_models.types import BotType + + +class ConnectedSubmind(BaseModel): + service_name: str = Field( + description="Name of the submind service (not its UID)") + attached_cids: List[str] = Field( + default=[], alias="cids", + description="List of conversation IDs the submind is participating in") + version: Optional[str] = Field( + default=None, + description="Version of chatbot-core the submind is using") + supports_raw_conversation: bool = Field( + default=False, + description="True if the submind will handle all conversation shouts") + last_ping: datetime = Field( + default_factory=lambda: datetime.now(tz=timezone.utc), + description="Last time the submind pinged the observer") + + bot_type: BotType = Field( + deprecated=True, default="submind", + description="Type of bot (always `submind` in this context)") + + +__all__ = [ConnectedSubmind.__name__] diff --git a/neon_data_models/models/api/llm.py b/neon_data_models/models/api/llm.py index a967f462..d78119ca 100644 --- a/neon_data_models/models/api/llm.py +++ b/neon_data_models/models/api/llm.py @@ -36,7 +36,8 @@ class LLMPersonaIdentity(BaseModel): """ Defines metadata for a unique persona. """ - name: str = Field(description="Unique name for this persona") + name: str = Field(alias="persona_name", + description="Unique name for this persona") user_id: Optional[str] = Field( None, description="`user_id` of the user who created this persona.") diff --git a/neon_data_models/models/api/mq/__init__.py b/neon_data_models/models/api/mq/__init__.py index 06db0711..da32c610 100644 --- a/neon_data_models/models/api/mq/__init__.py +++ b/neon_data_models/models/api/mq/__init__.py @@ -24,6 +24,7 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +from neon_data_models.models.api.mq.chatbots import * from neon_data_models.models.api.mq.llm import * -from neon_data_models.models.api.mq.users import * from neon_data_models.models.api.mq.neon import * +from neon_data_models.models.api.mq.users import * diff --git a/neon_data_models/models/api/mq/chatbots.py b/neon_data_models/models/api/mq/chatbots.py new file mode 100644 index 00000000..de81601a --- /dev/null +++ b/neon_data_models/models/api/mq/chatbots.py @@ -0,0 +1,446 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2025 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from typing import Any, Dict, Literal, Optional, List, Union +from datetime import datetime, timezone +from pydantic import Field, model_validator + +from neon_data_models.enum import SubmindStatus, CcaiState, CcaiControl +from neon_data_models.types import BotType +from neon_data_models.models.api.llm import LLMPersona +from neon_data_models.models.api.chatbots import ConnectedSubmind +from neon_data_models.models.base import BaseModel +from neon_data_models.models.base.contexts import KlatContext, MQContext + + +class ChatbotsMqRequest(KlatContext, MQContext): + """ + Defines a request from Klat to the Chatbots service. + """ + username: str = Field(alias="nick", + description="Username (or 'nick') of the sender") + cid: str = Field(description="Conversation ID associated with the shout") + message_text: str = Field(alias="messageText", + description="Text content of the shout") + from_bot: bool = Field( + default=False, + description="True if the shout is from a bot, False if from a user") + prompt_id: Optional[str] = Field( + default=None, + description="ID of the CCAI prompt associated with the shout") + prompt_state: Optional[CcaiState] = Field( + default=None, deprecated=True, + description="State of the CCAI conversation associated with the shout") + time_created: datetime = Field( + default= datetime.now(tz=timezone.utc), + description="Timestamp when the shout was created") + requested_participants: Optional[List[str]] = Field( + default=None, + description="List of CCAI participants requested to handle the shout") + recipient: Optional[str] = Field( + default=None, description="Explicitly defined recipient of the shout. ") + bound_service: Optional[str] = Field( + default=None, description="Service bound to the conversation") + + @classmethod + def from_sio_message(cls, sio_message: dict) -> 'ChatbotsMqRequest': + klat_context = KlatContext(**sio_message) + mq_context = MQContext(**sio_message) + return ChatbotsMqRequest( + **klat_context.model_dump(exclude_none=True), + **mq_context.model_dump(exclude_none=True), + username=sio_message.get("userDisplayName") or \ + sio_message.get("userID"), + message_text=sio_message["messageText"], + from_bot=sio_message.get("bot") == 1, + prompt_id = sio_message.get("promptID"), + prompt_state=sio_message.get("promptState"), + time_created=sio_message["timeCreated"], + recipient=sio_message.get("recipient"), + bound_service=sio_message.get("bound_service"), + ) + + def model_dump(self, **kwargs): + """ + Override model_dump to include SIO fields for backwards compatibility + """ + + # For backwards-compat with Klat Server, include aliased keys in + # serialization. In the future, this should be configurable and + # eventually removed. + by_alias = {} + if 'by_alias' not in kwargs: + by_alias = super().model_dump(by_alias=True, **kwargs) + + # Add parameters for backwards-compat. + by_alias["bot"] = "1" if self.from_bot else "0" + + return {**super().model_dump(**kwargs), **by_alias} + + +class ChatbotsMqSubmindResponse(KlatContext, MQContext): + """ + Defines a chatbot response to a request. + """ + user_id: str = Field(alias='userID', + description="Unique UID of the sender") + username: Optional[str] = Field(default=None, + alias="userDisplayName", + description="Username of the sender") + message_text: str = Field(alias="messageText", + description="Text content of the shout") + sid: str = Field(default="", alias="messageID", description="Shout ID") + replied_message: Optional[str] = Field( + default=None, alias="repliedMessage", + description="ID of the shout being replied to") + bot: Literal["0", "1"] = Field(default='0', alias='is_bot', + description="1 if the shout is from a bot") + prompt_id: Optional[str] = Field( + default=None, alias="promptID", + description="ID of the CCAI prompt associated with the shout") + is_announcement: bool = Field( + default=False, alias="isAnnouncement", + description="True if the shout is an announcement") + time_created: datetime = Field( + default= datetime.now(tz=timezone.utc), alias="timeCreated", + description="Timestamp when the shout was created") + source: str = Field( + default="klat_observer", + description="Name of the service originating the shout") + bot_type: Optional[BotType] = Field(default=None, deprecated=True, + description="Type of submind sending the shout") + + # Below are deprecated fields for backwards-compat. + service_name: Any = Field(default=None, deprecated=True) + context: Optional[dict] = Field( + default=None, deprecated=True, + description="Context used for Klat Server backwards-compat.") + dom: Any = Field(default=None, deprecated=True, + description="Domain of this conversation") + omit_reply: bool = Field( + default=True, deprecated=True, + description="If true, the Proctor will ignore this message") + no_save: bool = Field(default=False, deprecated=True, + description="If true, this message will be ignored") + to_discussion: bool = Field(default=False, deprecated=True) + prompt_state: CcaiState = Field( + default=CcaiState.IDLE, deprecated=True, alias="promptState", + description="State of the CCAI conversation associated with the shout") + + @model_validator(mode='before') + @classmethod + def validate_inputs(cls, values): + if isinstance(values, dict): + # Some additional aliases for backwards-compat. + if "nick" in values: + values.setdefault("userID", values.get("nick")) + + if "shout" in values: + values.setdefault("messageText", values.get("shout")) + + if "responded_shout" in values: + values.setdefault("repliedMessage", + values.get("responded_shout")) + + if "time" in values: + values.setdefault("timeCreated", values.get("time")) + + if "created_on" in values: + values.setdefault("timeCreated", values.get("created_on")) + + if "sid" in values and values["sid"] is None: + values.pop("sid") + + if "conversation_state" in values: + values.setdefault("promptState", + values.get("conversation_state")) + + return values + + def model_dump(self, **kwargs): + # For backwards-compat with Klat Server, include aliased keys in + # serialization. In the future, this should be configurable and + # eventually removed. + by_alias = {} + if 'by_alias' not in kwargs: + by_alias = super().model_dump(by_alias=True, **kwargs) + + by_alias['isAnnouncement'] = '1' if self.is_announcement else '0' + by_alias['nick'] = self.user_id + by_alias['responded_shout'] = self.replied_message + by_alias['shout'] = self.message_text + by_alias['time'] = self.time_created.timestamp() + by_alias['promptState'] = self.prompt_state.value + by_alias['created_on'] = self.time_created.timestamp() + + return {**super().model_dump(**kwargs), **by_alias} + + +class PromptCompletedContext(BaseModel): + prompt: Optional[ChatbotsMqRequest] = Field( + default=None, description="Original request containing the prompt") + + prompt_text: str = Field(description="The string prompt that is completed") + available_subminds: List[str] = Field( # Seems to always match `participating_subminds` + default=[], description="List of subminds available to participate") + participating_subminds: List[str] = Field( + default=[], description="List of subminds participating in the prompt") + proposed_responses: Dict[str, str] = Field( + default={}, description="Dict of nick to proposal") + + # In the future, there will be a list of these for multi-round discussion + submind_opinions: Dict[str, str] = Field( + default={}, description="Dict of nick to discussion") + + votes: Dict[str, str] = Field(default={}, + description="Dict of nick to vote") + votes_per_submind: Dict[str, List[str]] = Field( + default={}, description="Dict of nick to list of received votes") + winner: str = Field(default="", description="Selected winner") + + # Below are deprecated + is_active: bool = Field( # Seems to report active all the time + default=False, deprecated=True, + description="True if a response has not yet been chosen") + state: Optional[CcaiState] = Field( + default=CcaiState.PICK, deprecated=True, + description="State of the CCAI conversation (always PICK)") + + +class ChatbotsMqSavePrompt(ChatbotsMqSubmindResponse): + context: PromptCompletedContext = Field( + alias="conversation_context", + description="Definition of the completed discussion") + + def model_dump(self, **kwargs): + return ChatbotsMqSubmindResponse.model_dump(self, **kwargs) + + +class ChatbotsMqNewPrompt(ChatbotsMqSubmindResponse): + prompt_id: str = Field( + description="ID of the CCAI prompt associated with the shout" + ) + user_id: Optional[str] = Field(default=None, alias="nick", + description="User ID of the proctor") + prompt_text: str = Field(description="The new prompt being discussed") + prompt_state: CcaiState = Field( + default=CcaiState.IDLE, deprecated=True, + description="Implemented for backwards-compat. New Prompt always IDLE") + context: Optional[dict] = Field(default=None, deprecated=True, + alias="conversation_context") + + @model_validator(mode='before') + @classmethod + def validate_context(cls, values): + values.setdefault("message_text", values.get("shout", "")) + values.setdefault("user_id", values.get("nick")) + return values + + def model_dump(self, **kwargs): + return ChatbotsMqSubmindResponse.model_dump(self, **kwargs) + + +class ChatbotsMqResponse: + """ + Type adapter for validating an arbitrary MQ message. This will always return + an instance that extends `BaseMessage` and `MQContext`. + """ + @classmethod + def __new__(cls, *_, **kwargs) -> Union[ChatbotsMqSavePrompt, + ChatbotsMqNewPrompt, + ChatbotsMqSubmindResponse]: + message_text = kwargs.get("message_text") or kwargs.get("messageText") \ + or kwargs.get("shout") + kwargs['message_text'] = message_text + + if message_text == CcaiControl.SAVE_PROMPT_RESULTS.value: + return ChatbotsMqSavePrompt(**kwargs) + elif message_text == CcaiControl.CREATE_PROMPT.value: + return ChatbotsMqNewPrompt(**kwargs) + else: + return ChatbotsMqSubmindResponse(**kwargs) + + + + +class ChatbotsMqSubmindsState(MQContext): + class SubmindState(BaseModel): + submind_id: str = Field(description="Connected submind's user_id") + status: SubmindStatus = Field( + description="Subminds's status in a particular conversation") + + subminds_per_cid: Dict[str, List[SubmindState]] = Field( + description="List of submind participants per conversation ID") + connected_subminds: Dict[str, ConnectedSubmind] = Field( + description="Dict of submind `user_id` to `ConnectedSubmind` object") + cid_submind_bans: Dict[str, List[str]] = Field( + description="Dict of `cid` to list of banned submind `user_id`s") + banned_subminds: List[str] = Field( + description="List of globally banned submind `user_id`s") + + msg_type: Literal["subminds_state"] = Field( + "subminds_state", description="Message type for SIO", deprecated=True) + + +class ChatbotsMqConfiguredPersonasRequest(MQContext): + service_name: str = Field( + description="Name of the service to get personas for") + user_id: Optional[str] = Field( + default=None, description="Optional user_id making with the request.") + + +class ChatbotsMqConfiguredPersonasResponse(MQContext): + update_time: datetime = Field( + description="Time the personas were last checked") + items: List[LLMPersona] = Field( + description="List of configured personas from Klat") + + context: dict = Field(deprecated=True) + + @model_validator(mode='before') + @classmethod + def validate_context(cls, values): + # Deprecated context handling for backwards-compat. + if 'context' not in values and 'message_id' in values: + values['context'] = {"mq": {"message_id": values['message_id']}} + return values + + def model_dump(self, **kwargs): + """ + Override model_dump to include 'persona_name' field for each item based + on its 'name' for backwards-compat. with Klat server + """ + by_alias = {} + if 'by_alias' not in kwargs: + # `by_alias` to include `persona_name` in serialized `LLMPersona`s + by_alias = super().model_dump(by_alias=True, **kwargs) + + return {**super().model_dump(**kwargs), **by_alias} + + @classmethod + def from_persona_request(cls, data: dict, + request: ChatbotsMqConfiguredPersonasRequest): + data["items"] = [item for item in data["items"] + if request.service_name in item["supported_llms"]] + return cls(**data, message_id=request.message_id, + routing_key=request.routing_key) + + +class ChatbotsMqPromptsDataRequest(MQContext): + """ + Convenience class. The message payload here is just `MQContext`. + """ + + +class ChatbotsMqPromptsDataResponse(MQContext): + records: List[str] = Field(description="List of configured prompts") + + context: dict = Field(deprecated=True) + + @model_validator(mode='before') + @classmethod + def validate_context(cls, values): + # Deprecated context handling for backwards-compat. + if 'context' not in values and 'message_id' in values: + values['context'] = {"mq": {"message_id": values['message_id']}} + return values + + @classmethod + def from_prompt_data_request(cls, data: dict, + request: ChatbotsMqPromptsDataRequest): + return cls(**data, message_id=request.message_id, + routing_key=request.routing_key) + + +class ChatbotsMqSubmindConnection(MQContext): + user_id: str = Field(description="User ID of the submind", alias="nick") + time: datetime = Field( + default= datetime.now(tz=timezone.utc), + description="Timestamp when the submind last connected") + cids: Optional[List[str]] = Field( + default=None, description="List of conversation IDs the submind is in") + context: Optional[ConnectedSubmind] = Field( + default=None, + description="ConnectedSubmind definition of the connecting bot") + + @model_validator(mode='before') + @classmethod + def validate_context(cls, values): + if "context" in values and isinstance(values["context"], dict): + user_id = values.get("user_id") or values.get("nick") or "" + values["context"].setdefault("service_name", + user_id.rsplit('-',1)[0]) + return values + + +class ChatbotsMqSubmindDisconnection(MQContext): + user_id: str = Field(description="User ID of the submind", alias="nick") + + +class ChatbotsMqSubmindInvitation(MQContext): + cid: str = Field(description="Conversation ID to invite subminds to") + requested_participants: List[str] = Field( + description="List of submind User IDs to invite to the conversation") + + +class ChatbotsMqUpdateParticipatingSubminds(MQContext): + cid: str = Field(description="Conversation ID to update") + subminds_to_invite: List[str] = Field( + default=[], + description="List of submind User IDs to invite to the conversation") + subminds_to_kick: List[str] = Field( + default=[], + description="List of submind User IDs to evict from the conversation") + + +class ChatbotsMqSubmindConversationBan(MQContext): + user_id: str = Field(description="User ID of the submind", alias="nick") + cid: str = Field(description="Conversation ID to (un)ban submind from") + + +class ChatbotsMqSubmindGlobalBan(MQContext): + user_id: str = Field(description="User ID of the submind", alias="nick") + + +class ChatbotsMqSubmindResponseError(MQContext): + message: Optional[str] = Field(default=None, alias="msg", + description="Error message") + + +__all__ = [ChatbotsMqRequest.__name__, ChatbotsMqResponse.__name__, + ChatbotsMqSubmindResponse.__name__, ChatbotsMqSavePrompt.__name__, + ChatbotsMqNewPrompt.__name__, ChatbotsMqSubmindsState.__name__, + ChatbotsMqConfiguredPersonasRequest.__name__, + ChatbotsMqConfiguredPersonasResponse.__name__, + ChatbotsMqPromptsDataRequest.__name__, + ChatbotsMqPromptsDataResponse.__name__, + ChatbotsMqSubmindConnection.__name__, + ChatbotsMqSubmindDisconnection.__name__, + ChatbotsMqSubmindInvitation.__name__, + ChatbotsMqUpdateParticipatingSubminds.__name__, + ChatbotsMqSubmindConversationBan.__name__, + ChatbotsMqSubmindGlobalBan.__name__, + ChatbotsMqSubmindResponseError.__name__,] diff --git a/neon_data_models/models/base/__init__.py b/neon_data_models/models/base/__init__.py index 220c7e3b..b31cba13 100644 --- a/neon_data_models/models/base/__init__.py +++ b/neon_data_models/models/base/__init__.py @@ -24,6 +24,7 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +from enum import Enum from os import environ from datetime import datetime, timedelta from pydantic import ConfigDict, BaseModel as _BaseModel @@ -31,8 +32,9 @@ class BaseModel(_BaseModel): model_config = ConfigDict(extra="allow" if environ.get( - "NEON_DATA_MODELS_ALLOW_EXTRA", "false") != "false" else "ignore") - + "NEON_DATA_MODELS_ALLOW_EXTRA", "false") != "false" else "ignore", + populate_by_name=environ.get("NEON_DATA_MODELS_POPULATE_BY_NAME", + "true") != "false",) def model_dump(self, *args, **kwargs) -> dict: """ @@ -60,6 +62,9 @@ def _process_data(self, data): elif isinstance(data, timedelta): # Convert timedelta to seconds return data.total_seconds() + elif isinstance(data, Enum): + # Always serialize `Enum` objects by value + return data.value else: # Return other types unchanged return data diff --git a/neon_data_models/models/base/contexts.py b/neon_data_models/models/base/contexts.py index c944514e..1a560191 100644 --- a/neon_data_models/models/base/contexts.py +++ b/neon_data_models/models/base/contexts.py @@ -26,6 +26,7 @@ from datetime import datetime, timedelta, timezone from typing import Literal, List, Optional, Any, Tuple +from uuid import uuid4 from pydantic import Field, model_validator @@ -136,7 +137,8 @@ def validate_inputs(cls, values): class MQContext(BaseModel): routing_key: Optional[str] = None - message_id: str + message_id: str = Field(default_factory=lambda: uuid4().hex, + description="MQ unique message ID") @model_validator(mode='before') @classmethod @@ -145,4 +147,7 @@ def validate_inputs(cls, values): Validate MQContext inputs to normalize messageID to message_id. """ values.setdefault("message_id", values.get("messageID")) + if values['message_id'] is None: + # Allow default_factory to generate a message_id + values.pop("message_id") return values diff --git a/neon_data_models/types.py b/neon_data_models/types.py index c656b6ad..62b9a7dd 100644 --- a/neon_data_models/types.py +++ b/neon_data_models/types.py @@ -28,3 +28,5 @@ Gender = Literal["male", "female"] +BotType = Literal["submind", "facilitator", "proctor", "observer"] +# TODO: consider deprecating special facilitator types in `BotType` diff --git a/tests/models/api/test_chatbots.py b/tests/models/api/test_chatbots.py new file mode 100644 index 00000000..089459d9 --- /dev/null +++ b/tests/models/api/test_chatbots.py @@ -0,0 +1,73 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2025 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from unittest import TestCase +from datetime import datetime, timezone +from pydantic import ValidationError + + +class TestChatbots(TestCase): + def test_connected_submind(self): + from neon_data_models.models.api.chatbots import ConnectedSubmind + + # Test minimal creation + minimal_submind = ConnectedSubmind(service_name="test_service") + self.assertEqual(minimal_submind.service_name, "test_service") + self.assertEqual(minimal_submind.attached_cids, []) + self.assertIsNone(minimal_submind.version) + self.assertFalse(minimal_submind.supports_raw_conversation) + self.assertIsInstance(minimal_submind.last_ping, datetime) + self.assertEqual(minimal_submind.bot_type, "submind") + + # Test full definition + full_submind = ConnectedSubmind( + service_name="full_service", + attached_cids=["cid1", "cid2"], + version="1.0.0", + supports_raw_conversation=True + ) + self.assertEqual(full_submind.service_name, "full_service") + self.assertEqual(full_submind.attached_cids, ["cid1", "cid2"]) + self.assertEqual(full_submind.version, "1.0.0") + self.assertTrue(full_submind.supports_raw_conversation) + + # Test alias + cids_submind = ConnectedSubmind( + service_name="cids_service", + cids=["cid3"] + ) + self.assertEqual(cids_submind.attached_cids, ["cid3"]) + + # Test validation + with self.assertRaises(ValidationError): + ConnectedSubmind() # Missing required field service_name + + # Test datetime factory + import time + submind1 = ConnectedSubmind(service_name="time_test") + time.sleep(0.1) + submind2 = ConnectedSubmind(service_name="time_test_2") + self.assertLess(submind1.last_ping, submind2.last_ping) diff --git a/tests/models/api/test_llm.py b/tests/models/api/test_llm.py index af7e902c..3ae9b0ee 100644 --- a/tests/models/api/test_llm.py +++ b/tests/models/api/test_llm.py @@ -37,6 +37,11 @@ def test_llm_persona(self): self.assertEqual(legacy_mq_persona.system_prompt, legacy_mq_persona.description) + # By alias + alias_mq_persona = LLMPersona(persona_name="my persona", + description="You are a helpful chatbot.") + self.assertEqual(alias_mq_persona, legacy_mq_persona) + # Valid system prompt legacy_bf_persona = LLMPersona(name="neon", system_prompt="You are NeonLLM.") diff --git a/tests/models/api/test_mq.py b/tests/models/api/test_mq.py index c4e33c6a..14a62038 100644 --- a/tests/models/api/test_mq.py +++ b/tests/models/api/test_mq.py @@ -26,6 +26,7 @@ from unittest import TestCase from pydantic import ValidationError +from datetime import datetime, timezone class TestUsersMQ(TestCase): @@ -161,9 +162,9 @@ def test_mq_llm_propose_request(self): LLMProposeRequest(query=query, history=history, message_id=message_id, persona={}) - # Invalid MQ Context - with self.assertRaises(ValidationError): - LLMProposeRequest(query=query, history=history) + # # Invalid MQ Context + # with self.assertRaises(ValidationError): + # LLMProposeRequest(query=query, history=history) # Invalid LLM Request with self.assertRaises(ValidationError): @@ -177,9 +178,9 @@ def test_mq_llm_propose_response(self): message_id=""), LLMProposeResponse) - # Missing MQ required data - with self.assertRaises(ValidationError): - LLMProposeResponse(response="test response") + # # Missing MQ required data + # with self.assertRaises(ValidationError): + # LLMProposeResponse(response="test response") # Missing response required data with self.assertRaises(ValidationError): @@ -203,9 +204,9 @@ def test_mq_llm_discuss_request(self): LLMDiscussRequest(query=query, history=history, message_id=message_id, options=invalid_opts) - # Invalid MQ Context - with self.assertRaises(ValidationError): - LLMDiscussRequest(query=query, history=history, options=opts) + # # Invalid MQ Context + # with self.assertRaises(ValidationError): + # LLMDiscussRequest(query=query, history=history, options=opts) # Invalid LLM Request with self.assertRaises(ValidationError): @@ -219,9 +220,9 @@ def test_mq_llm_discuss_response(self): message_id=""), LLMDiscussResponse) - # Missing MQ required data - with self.assertRaises(ValidationError): - LLMDiscussResponse(opinion="test opinion") + # # Missing MQ required data + # with self.assertRaises(ValidationError): + # LLMDiscussResponse(opinion="test opinion") # Missing response required data with self.assertRaises(ValidationError): @@ -246,9 +247,9 @@ def test_mq_llm_vote_request(self): LLMVoteRequest(query=query, history=history, message_id=message_id, responses=invalid_responses) - # Invalid MQ Context - with self.assertRaises(ValidationError): - LLMVoteRequest(query=query, history=history, responses=responses) + # # Invalid MQ Context + # with self.assertRaises(ValidationError): + # LLMVoteRequest(query=query, history=history, responses=responses) # Invalid LLM Request with self.assertRaises(ValidationError): @@ -263,9 +264,9 @@ def test_mq_llm_vote_response(self): message_id=""), LLMVoteResponse) - # Missing MQ required data - with self.assertRaises(ValidationError): - LLMVoteResponse(sorted_answer_indexes=[2, 0, 1]) + # # Missing MQ required data + # with self.assertRaises(ValidationError): + # LLMVoteResponse(sorted_answer_indexes=[2, 0, 1]) # Missing response required data with self.assertRaises(ValidationError): @@ -794,3 +795,1313 @@ def test_neon_api_message_with_nested_data(self): self.assertEqual(message.data["list_data"][3]["key"], "value") self.assertEqual(message.context.client, "test_client") + +class TestChatbotsMQ(TestCase): + def test_chatbots_mq_request(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqRequest + + # Test basic initialization with required fields + current_time = datetime.now(tz=timezone.utc) + valid_kwargs = { + "username": "test_user", + "cid": "test_conversation", + "message_text": "Hello, how are you?", + "message_id": "test_message_id", + "time_created": current_time + } + + chatbot_request = ChatbotsMqRequest(**valid_kwargs) + self.assertIsInstance(chatbot_request, ChatbotsMqRequest) + self.assertEqual(chatbot_request.username, "test_user") + self.assertEqual(chatbot_request.cid, "test_conversation") + self.assertEqual(chatbot_request.message_text, "Hello, how are you?") + self.assertEqual(chatbot_request.time_created, current_time) + self.assertFalse(chatbot_request.from_bot) # Default value check + + # Test with all fields + full_kwargs = { + "username": "test_user", + "cid": "test_conversation", + "message_text": "Hello, how are you?", + "from_bot": True, + "prompt_id": "test_prompt_id", + "prompt_state": 1, + "time_created": current_time, + "requested_participants": ["participant1", "participant2"], + "recipient": "test_recipient", + "bound_service": "test_service", + "client": "test_client", + "message_id": "test_message_id", + "messageID": "test_sid" + } + + full_request = ChatbotsMqRequest(**full_kwargs) + self.assertIsInstance(full_request, ChatbotsMqRequest) + self.assertEqual(full_request.username, "test_user") + self.assertEqual(full_request.cid, "test_conversation") + self.assertEqual(full_request.sid, "test_sid") + self.assertEqual(full_request.message_text, "Hello, how are you?") + self.assertTrue(full_request.from_bot) + self.assertEqual(full_request.prompt_id, "test_prompt_id") + self.assertEqual(full_request.prompt_state, 1) + self.assertEqual(full_request.time_created, current_time) + self.assertEqual(full_request.requested_participants, ["participant1", "participant2"]) + self.assertEqual(full_request.recipient, "test_recipient") + self.assertEqual(full_request.bound_service, "test_service") + self.assertEqual(full_request.message_id, "test_message_id") + + # Test missing required fields + with self.assertRaises(ValidationError): + # Missing message text + ChatbotsMqRequest(username="test_user", cid="test_conversation") + + with self.assertRaises(ValidationError): + # Missing username + ChatbotsMqRequest(cid="test_conversation", message_text="Hello", + time_created=current_time) + + with self.assertRaises(ValidationError): + # Missing cid + ChatbotsMqRequest(username="test_user", message_text="Hello", time_created=current_time) + + # Test model_dump method for backwards compatibility + # Test with from_bot=True + bot_request = ChatbotsMqRequest( + username="test_user", + cid="test_conversation", + message_id="test_message_id", + message_text="Hello from bot", + from_bot=True, + time_created=datetime.now(tz=timezone.utc) + ) + serialized_bot = bot_request.model_dump() + self.assertEqual(serialized_bot["bot"], "1") + self.assertTrue(serialized_bot["from_bot"]) + self.assertEqual(serialized_bot["messageText"], bot_request.message_text) + self.assertEqual(serialized_bot["nick"], bot_request.username) + + # Test with from_bot=False (default) + user_request = ChatbotsMqRequest( + username="test_user", + cid="test_conversation", + message_id="test_message_id", + message_text="Hello from user", + time_created=datetime.now(tz=timezone.utc) + ) + serialized_user = user_request.model_dump() + self.assertEqual(serialized_user["bot"], "0") + self.assertFalse(serialized_user["from_bot"]) + + # Test from_sio_message method + sio_message = { + "userDisplayName": "display_name", + "userID": "user_id", + "cid": "test_conversation", + "messageText": "Hello from SIO", + "bot": 1, + "promptID": "test_prompt_id", + "promptState": 2, + "timeCreated": current_time, + "recipient": "chatbots", + "bound_service": "test_service", + "message_id": "test_message_id" + } + + sio_request = ChatbotsMqRequest.from_sio_message(sio_message) + self.assertIsInstance(sio_request, ChatbotsMqRequest) + self.assertEqual(sio_request.username, "display_name") + self.assertEqual(sio_request.cid, "test_conversation") + self.assertEqual(sio_request.message_text, "Hello from SIO") + self.assertTrue(sio_request.from_bot) + self.assertEqual(sio_request.prompt_id, "test_prompt_id") + self.assertEqual(sio_request.prompt_state, 2) + self.assertEqual(sio_request.time_created, current_time) + self.assertEqual(sio_request.recipient, "chatbots") + self.assertEqual(sio_request.bound_service, "test_service") + + # Test userID fallback + sio_message = { + "userID": "user_id", + "cid": "test_conversation", + "messageText": "Hello from SIO", + "timeCreated": current_time, + "message_id": "test_message_id" + } + + sio_request = ChatbotsMqRequest.from_sio_message(sio_message) + self.assertEqual(sio_request.username, "user_id") + + def test_chatbots_mq_submind_response(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqResponse, ChatbotsMqSubmindResponse + + # Test basic initialization with required fields + current_time = datetime.now(tz=timezone.utc) + valid_kwargs = { + "userID": "user123", + "messageText": "Hello, this is a response", + "message_id": "test_message_id", + "conversation_state": 0 + } + + response = ChatbotsMqResponse(**valid_kwargs) + self.assertIsInstance(response, ChatbotsMqSubmindResponse) + self.assertEqual(response.user_id, "user123") + self.assertEqual(response.message_text, "Hello, this is a response") + self.assertEqual(response.bot, "0") # Default value check + self.assertFalse(response.is_announcement) # Default value check + + # Test with all fields including aliases + full_kwargs = { + "userID": "user123", + "userDisplayName": "John Doe", + "messageText": "Hello, this is a complete response", + "messageID": "msg123", + "repliedMessage": "original_msg456", + "bot": "1", + "promptID": "prompt789", + "prompt_state": 2, + "is_announcement": True, + "timeCreated": current_time, + "source": "test_source", + "client": "test_client", + "cid": "conversation123", + "message_id": "message123" + } + + full_response = ChatbotsMqResponse(**full_kwargs) + self.assertIsInstance(full_response, ChatbotsMqSubmindResponse) + self.assertEqual(full_response.user_id, "user123") + self.assertEqual(full_response.username, "John Doe") + self.assertEqual(full_response.message_text, "Hello, this is a complete response") + self.assertEqual(full_response.sid, "msg123") + self.assertEqual(full_response.replied_message, "original_msg456") + self.assertEqual(full_response.bot, "1") + self.assertEqual(full_response.prompt_id, "prompt789") + self.assertEqual(full_response.prompt_state, 2) + self.assertTrue(full_response.is_announcement) + self.assertEqual(full_response.time_created, current_time) + self.assertEqual(full_response.source, "test_source") + self.assertEqual(full_response.cid, "conversation123") + self.assertEqual(full_response.message_id, "message123") + + # Test model_dump with aliased fields + serialized = full_response.model_dump() + self.assertIn("userID", serialized) + self.assertIn("userDisplayName", serialized) + self.assertIn("messageText", serialized) + self.assertIn("messageID", serialized) + self.assertIn("repliedMessage", serialized) + self.assertIn("promptID", serialized) + self.assertIn("isAnnouncement", serialized) + self.assertIn("timeCreated", serialized) + + # Test invalid literal values + with self.assertRaises(ValidationError): + ChatbotsMqResponse( + userID="user123", + messageText="Test", + bot="2", # Invalid value, only "0" or "1" allowed + message_id="test_id" + ) + + with self.assertRaises(ValidationError): + ChatbotsMqResponse( + userID="user123", + messageText="Test", + isAnnouncement="2", # Invalid value, only "0" or "1" allowed + message_id="test_id" + ) + + def test_prompt_completed_context(self): + from neon_data_models.models.api.mq.chatbots import PromptCompletedContext, ChatbotsMqRequest + from datetime import datetime, timezone + + # Create a test ChatbotsMqRequest for the prompt field + request = ChatbotsMqRequest( + username="test_user", + cid="test_conversation", + message_text="Test prompt", + message_id="test_message_id", + time_created=datetime.now(tz=timezone.utc) + ) + + # Test initialization with required fields + valid_kwargs = { + "prompt": request, + "is_active": True, + "prompt_text": "Test prompt text", + "available_subminds": ["submind1", "submind2"], + "state": 1, + "participating_subminds": ["submind1"], + "proposed_responses": {"submind1": "Response 1"}, + "submind_opinions": {"submind1": "Opinion 1"}, + "votes": {"submind1": "vote1"}, + "votes_per_submind": {"submind1": ["vote1"]} + } + + context = PromptCompletedContext(**valid_kwargs) + self.assertIsInstance(context, PromptCompletedContext) + self.assertEqual(context.prompt, request) + self.assertTrue(context.is_active) + self.assertEqual(context.prompt_text, "Test prompt text") + self.assertEqual(context.available_subminds, ["submind1", "submind2"]) + self.assertEqual(context.state, 1) + self.assertEqual(context.participating_subminds, ["submind1"]) + self.assertEqual(context.proposed_responses, {"submind1": "Response 1"}) + self.assertEqual(context.submind_opinions, {"submind1": "Opinion 1"}) + self.assertEqual(context.votes, {"submind1": "vote1"}) + self.assertEqual(context.votes_per_submind, {"submind1": ["vote1"]}) + self.assertEqual(context.winner, "") # Default value + + # Test with winner field + context_with_winner = PromptCompletedContext( + **valid_kwargs, + winner="submind1" + ) + self.assertEqual(context_with_winner.winner, "submind1") + + # Test missing required fields + with self.assertRaises(ValidationError): + PromptCompletedContext( + available_subminds=["submind1", "submind2"], + participating_subminds=["submind1"], + proposed_responses={"submind1": "Response 1"}, + submind_opinions={"submind1": "Opinion 1"}, + votes={"submind1": "vote1"}, + votes_per_submind={"submind1": ["vote1"]} + ) + + def test_chatbots_mq_save_prompt(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSavePrompt, PromptCompletedContext, ChatbotsMqRequest + from datetime import datetime, timezone + + # Create necessary components for testing + request = ChatbotsMqRequest( + username="test_user", + cid="test_conversation", + message_text="Test prompt", + message_id="test_message_id", + time_created=datetime.now(tz=timezone.utc) + ) + + context = PromptCompletedContext( + prompt=request, + is_active=False, + prompt_text="Test prompt text", + available_subminds=["submind1", "submind2"], + state=2, + participating_subminds=["submind1"], + proposed_responses={"submind1": "Response 1"}, + submind_opinions={"submind1": "Opinion 1"}, + votes={"submind1": "vote1"}, + votes_per_submind={"submind1": ["vote1"]}, + winner="submind1" + ) + + # Test initialization + valid_kwargs = { + "userID": "user123", + "messageText": "Prompt completed", + "message_id": "test_message_id", + "prompt_id": "prompt123", + "prompt_text": "Test prompt text", + "created_on": "2023-01-01", + "conversation_state": 0, + "context": context + } + save_prompt = ChatbotsMqSavePrompt(**valid_kwargs) + self.assertIsInstance(save_prompt, ChatbotsMqSavePrompt) + self.assertEqual(save_prompt.user_id, "user123") + self.assertEqual(save_prompt.message_text, "Prompt completed") + self.assertEqual(save_prompt.prompt_id, "prompt123") + # self.assertEqual(save_prompt.prompt_text, "Test prompt text") + # self.assertEqual(save_prompt.created_on, "2023-01-01") + self.assertEqual(save_prompt.context, context) + + # Test model_dump inheritance from ChatbotsMqResponse + serialized = save_prompt.model_dump() + self.assertIn("userID", serialized) + self.assertIn("messageText", serialized) + self.assertIn("prompt_id", serialized) + self.assertIn("context", serialized) + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqSavePrompt( + userID="user123", + messageText="Prompt completed", + message_id="test_message_id", + prompt_id="prompt123", + prompt_text="Test prompt text", + created_on="2023-01-01" + # Missing context + ) + + def test_chatbots_mq_new_prompt(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqNewPrompt + + # Test initialization with required fields + valid_kwargs = { + "user_id": "user123", + "messageText": "New prompt", + "message_id": "test_message_id", + "prompt_id": "prompt123", + "prompt_text": "Test prompt text", + "conversation_state": 0 + } + + new_prompt = ChatbotsMqNewPrompt(**valid_kwargs) + self.assertIsInstance(new_prompt, ChatbotsMqNewPrompt) + self.assertEqual(new_prompt.user_id, "user123") + self.assertEqual(new_prompt.message_text, valid_kwargs["messageText"]) + self.assertEqual(new_prompt.prompt_id, "prompt123") + self.assertEqual(new_prompt.prompt_text, "Test prompt text") + self.assertIsNone(new_prompt.context) + + # Test with conversation_context alias + context_kwargs = { + "user_id": "user123", + "messageText": "New prompt with context", + "message_id": "test_message_id", + "prompt_id": "prompt123", + "prompt_text": "Test prompt text", + "conversation_state": 0, + "conversation_context": {"some_key": "some_value"} + } + + context_prompt = ChatbotsMqNewPrompt(**context_kwargs) + self.assertEqual(context_prompt.context, {"some_key": "some_value"}) + + # Test model validator + # Create a message with prompt_text but no messageText + partial_kwargs = { + "user_id": "user123", + "message_id": "test_message_id", + "prompt_id": "prompt123", + "prompt_text": "Only prompt text", + "conversation_state": 0 + } + partial_prompt = ChatbotsMqNewPrompt(**partial_kwargs) + self.assertEqual(partial_prompt.message_text, '') + + # Test model_dump inheritance from ChatbotsMqResponse + serialized = new_prompt.model_dump() + self.assertIn("user_id", serialized) + self.assertIn("messageText", serialized) + self.assertIn("prompt_id", serialized) + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqNewPrompt( + user_id="user123", + messageText="New prompt", + message_id="test_message_id" + # Missing prompt_id + ) + + with self.assertRaises(ValidationError): + ChatbotsMqNewPrompt( + user_id="user123", + message_id="test_message_id", + prompt_id="prompt123" + # Missing messageText or prompt_text + ) + + def test_old_class_names_compatibility(self): + """Test backward compatibility with old class names""" + import neon_data_models.models.api.mq.chatbots as chatbots_module + + # Check if the old class names are still available through __all__ + self.assertIn("ChatbotsMqRequest", chatbots_module.__all__) + self.assertIn("ChatbotsMqResponse", chatbots_module.__all__) + self.assertIn("ChatbotsMqSavePrompt", chatbots_module.__all__) + self.assertIn("ChatbotsMqNewPrompt", chatbots_module.__all__) + + # Import the actual classes to verify they exist + from neon_data_models.models.api.mq.chatbots import ChatbotsMqRequest, ChatbotsMqResponse, ChatbotsMqSubmindResponse + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSavePrompt, ChatbotsMqNewPrompt + + # Test instantiation with basic properties + req = ChatbotsMqRequest( + username="test_user", + cid="test_conversation", + message_text="Test message", + message_id="test_message_id", + conversation_state=0, + time_created=datetime.now(timezone.utc) + ) + self.assertIsInstance(req, ChatbotsMqRequest) + + resp = ChatbotsMqResponse( + userID="test_user", + messageText="Test response", + message_id="test_message_id", + conversation_state=0, + ) + self.assertIsInstance(resp, ChatbotsMqSubmindResponse) + + # These tests verify that the classes are properly defined and usable + # even with the new names + + def test_connected_submind(self): + from neon_data_models.models.api.mq.chatbots import ConnectedSubmind + from neon_data_models.enum import CcaiState + from neon_data_models.types import BotType + + # Test valid initialization with required fields + current_time = datetime.now(tz=timezone.utc) + valid_kwargs = { + "message_id": "test_message_id", + "bot_type": "submind", + "service_name": "test_service", + "cid": "test_conversation", + "dom": "test_domain", + "conversation_state": CcaiState.IDLE, + "responded_shout": "test_shout_id", + "shout": "chatbot state", + "context": {"key": "value"}, + "prompt_id": "test_prompt_id", + "omit_reply": False, + "no_save": False, + "attached_cids": ["cid1", "cid2"], + "supports_raw_conversation": True, + "last_ping": current_time + } + + submind = ConnectedSubmind(**valid_kwargs) + self.assertIsInstance(submind, ConnectedSubmind) + self.assertIsInstance(submind.bot_type, str) + self.assertEqual(submind.service_name, "test_service") + self.assertEqual(submind.attached_cids, ["cid1", "cid2"]) + self.assertEqual(submind.last_ping, current_time) + self.assertTrue(submind.supports_raw_conversation) + + # Test model_dump functionality + serialized = submind.model_dump() + self.assertIsInstance(serialized["bot_type"], str) + self.assertEqual(serialized["service_name"], "test_service") + self.assertEqual(serialized["attached_cids"], ["cid1", "cid2"]) + + # Test validation logic for deprecated fields + deprecated_kwargs = valid_kwargs.copy() + deprecated_kwargs["bot_type"] = "proctor" + # deprecated_kwargs["shout"] = "hello" + submind = ConnectedSubmind(**deprecated_kwargs) + self.assertIsInstance(submind.bot_type, str) + # self.assertEqual(submind.shout, "chatbot state") + + def test_chatbots_mq_subminds_state(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindsState, ConnectedSubmind + from neon_data_models.enum import SubmindStatus + from neon_data_models.models.api.mq.chatbots import CcaiState + from datetime import datetime, timezone + + # Create test ConnectedSubmind for use in the test + current_time = datetime.now(tz=timezone.utc) + connected_submind = ConnectedSubmind( + message_id="connected_mid", + bot_type="submind", + service_name="test_service", + cid="test_conversation", + dom="test_domain", + conversation_state=CcaiState.IDLE, + responded_shout="test_shout_id", + shout="chatbot state", + context={"key": "value"}, + prompt_id="test_prompt_id", + omit_reply=False, + no_save=False, + attached_cids=["cid1", "cid2"], + supports_raw_conversation=True, + last_ping=current_time + ) + + # Create SubmindState objects for testing + submind_state1 = {"submind_id": "submind1", "status": SubmindStatus.ACTIVE} + submind_state2 = {"submind_id": "submind2", "status": SubmindStatus.BANNED} + + # Test valid initialization + valid_kwargs = { + "message_id": "test_message_id", + "subminds_per_cid": { + "cid1": [submind_state1, submind_state2], + "cid2": [submind_state1] + }, + "connected_subminds": { + "submind1": connected_submind + }, + "cid_submind_bans": { + "cid1": ["banned_submind1"], + "cid2": ["banned_submind2", "banned_submind3"] + }, + "banned_subminds": ["globally_banned1", "globally_banned2"] + } + + state = ChatbotsMqSubmindsState(**valid_kwargs) + self.assertIsInstance(state, ChatbotsMqSubmindsState) + + # Test the nested SubmindState objects + self.assertEqual(state.subminds_per_cid["cid1"][0].submind_id, "submind1") + self.assertEqual(state.subminds_per_cid["cid1"][0].status, SubmindStatus.ACTIVE) + self.assertEqual(state.subminds_per_cid["cid1"][1].submind_id, "submind2") + self.assertEqual(state.subminds_per_cid["cid1"][1].status, SubmindStatus.BANNED) + + # Test the connected_subminds mapping + self.assertIsInstance(state.connected_subminds["submind1"], ConnectedSubmind) + self.assertEqual(state.connected_subminds["submind1"].service_name, "test_service") + + # Test the ban lists + self.assertEqual(state.cid_submind_bans["cid1"], ["banned_submind1"]) + self.assertEqual(state.cid_submind_bans["cid2"], ["banned_submind2", "banned_submind3"]) + self.assertEqual(state.banned_subminds, ["globally_banned1", "globally_banned2"]) + + # Test model_dump functionality + serialized = state.model_dump() + self.assertIn("subminds_per_cid", serialized) + self.assertIn("connected_subminds", serialized) + self.assertIn("cid_submind_bans", serialized) + self.assertIn("banned_subminds", serialized) + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqSubmindsState( + message_id="test_message_id" + # Missing other required fields + ) + + def test_chatbots_mq_configured_personas_request(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqConfiguredPersonasRequest + + # Test valid initialization + valid_request = ChatbotsMqConfiguredPersonasRequest( + service_name="test_service", + message_id="test_message_id" + ) + self.assertIsInstance(valid_request, ChatbotsMqConfiguredPersonasRequest) + self.assertEqual(valid_request.service_name, "test_service") + self.assertEqual(valid_request.message_id, "test_message_id") + + # Test serialization + serialized = valid_request.model_dump() + self.assertEqual(serialized["service_name"], "test_service") + self.assertEqual(serialized["message_id"], "test_message_id") + + # Test missing service_name + with self.assertRaises(ValidationError): + ChatbotsMqConfiguredPersonasRequest(message_id="test_message_id") + + def test_chatbots_mq_configured_personas_response(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqConfiguredPersonasResponse + from neon_data_models.models.api.mq.chatbots import ChatbotsMqConfiguredPersonasRequest + from neon_data_models.models.api.llm import LLMPersona + + # Create test data + current_time = datetime.now(tz=timezone.utc) + test_personas = [ + LLMPersona( + name="persona1", + system_prompt="System prompt 1", + # supported_llms=["test_service", "another_service"] + ), + LLMPersona( + name="persona2", + system_prompt="System prompt 2", + # supported_llms=["test_service"] + ) + ] + + # Test valid initialization + valid_response = ChatbotsMqConfiguredPersonasResponse( + update_time=current_time, + items=test_personas, + message_id="test_message_id" + ) + self.assertIsInstance(valid_response, ChatbotsMqConfiguredPersonasResponse) + self.assertEqual(valid_response.update_time, current_time) + self.assertEqual(len(valid_response.items), 2) + self.assertEqual(valid_response.items[0].name, "persona1") + self.assertEqual(valid_response.items[1].name, "persona2") + + # Test model_dump behavior with persona_name addition + serialized = valid_response.model_dump() + self.assertEqual(serialized["items"][0]["persona_name"], "persona1") + self.assertEqual(serialized["items"][1]["persona_name"], "persona2") + + # Test from_persona_request method with ChatbotsMqConfiguredPersonasRequest + response_data = { + "update_time": current_time, + "items": [ + { + "name": "persona1", + "system_prompt": "System prompt 1", + "supported_llms": ["test_service", "another_service"] + }, + { + "name": "persona2", + "system_prompt": "System prompt 2", + "supported_llms": ["test_service"] + }, + { + "name": "persona3", + "system_prompt": "System prompt 3", + "supported_llms": ["another_service"] + } + ] + } + + # Create a request object + request = ChatbotsMqConfiguredPersonasRequest( + service_name="test_service", + message_id="test_req_id", + routing_key="test.routing.key", + user_id="test_user" + ) + + filtered_response = ChatbotsMqConfiguredPersonasResponse.from_persona_request( + response_data, request + ) + + # Verify filtering based on service_name + self.assertEqual(len(filtered_response.items), 2) # Only personas with "test_service" in supported_llms + self.assertEqual(filtered_response.items[0].name, "persona1") + self.assertEqual(filtered_response.items[1].name, "persona2") + + # Verify request properties are preserved + self.assertEqual(filtered_response.message_id, "test_req_id") + self.assertEqual(filtered_response.routing_key, "test.routing.key") + + # Test backward compatibility - omitting context + response_without_context = ChatbotsMqConfiguredPersonasResponse( + update_time=current_time, + items=test_personas, + message_id="test_message_id" + ) + self.assertIsInstance( + response_without_context.context['mq']['message_id'], str) + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqConfiguredPersonasResponse( + update_time=current_time, + items=test_personas + # Missing message_id + ) + + with self.assertRaises(ValidationError): + ChatbotsMqConfiguredPersonasResponse( + message_id="test_message_id", + items=test_personas + # Missing update_time + ) + + with self.assertRaises(ValidationError): + ChatbotsMqConfiguredPersonasResponse( + message_id="test_message_id", + update_time=current_time + # Missing items + ) + + def test_chatbots_mq_prompts_data_request(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqPromptsDataRequest + + # Test valid initialization + valid_request = ChatbotsMqPromptsDataRequest(message_id="test_message_id") + self.assertIsInstance(valid_request, ChatbotsMqPromptsDataRequest) + self.assertEqual(valid_request.message_id, "test_message_id") + + # # Test missing required fields + # with self.assertRaises(ValidationError): + # ChatbotsMqPromptsDataRequest() + + def test_chatbots_mq_prompts_data_response(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqPromptsDataResponse, ChatbotsMqPromptsDataRequest + + # Test valid initialization + valid_response = ChatbotsMqPromptsDataResponse( + message_id="test_message_id", records=["prompt1", "prompt2"] + ) + self.assertIsInstance(valid_response, ChatbotsMqPromptsDataResponse) + self.assertEqual(valid_response.records, ["prompt1", "prompt2"]) + + # Test from_prompt_data_request method + request = ChatbotsMqPromptsDataRequest(message_id="test_message_id") + response_data = {"records": ["prompt1", "prompt2"]} + response = ChatbotsMqPromptsDataResponse.from_prompt_data_request( + response_data, request + ) + self.assertEqual(response.message_id, "test_message_id") + self.assertEqual(response.records, ["prompt1", "prompt2"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqPromptsDataResponse(records=["prompt1", "prompt2"]) + + def test_chatbots_mq_submind_response(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindResponse + from datetime import datetime, timezone + + # Test basic initialization with required fields + current_time = datetime.now(tz=timezone.utc) + valid_kwargs = { + "userID": "test_user", + "messageText": "Test submind response", + "message_id": "test_message_id", + "prompt_state": 1, + } + + response = ChatbotsMqSubmindResponse(**valid_kwargs) + self.assertIsInstance(response, ChatbotsMqSubmindResponse) + self.assertEqual(response.user_id, "test_user") + self.assertEqual(response.message_text, "Test submind response") + self.assertEqual(response.bot, "0") # Default value check + + # Test with all fields including aliased fields + full_kwargs = { + "userID": "submind1", + "userDisplayName": "Test Submind", + "messageText": "Full test response", + "messageID": "test_shout_id", + "repliedMessage": "original_message_id", + "bot": "1", + "promptID": "prompt_id_123", + "conversation_state": 2, + "is_announcement": True, + "timeCreated": current_time, + "source": "test_source", + "client": "test_client", + "cid": "conversation123", + "message_id": "test_message_id" + } + + full_response = ChatbotsMqSubmindResponse(**full_kwargs) + self.assertIsInstance(full_response, ChatbotsMqSubmindResponse) + self.assertEqual(full_response.user_id, "submind1") + self.assertEqual(full_response.username, "Test Submind") + self.assertEqual(full_response.message_text, "Full test response") + self.assertEqual(full_response.sid, "test_shout_id") + self.assertEqual(full_response.replied_message, "original_message_id") + self.assertEqual(full_response.bot, "1") + self.assertEqual(full_response.prompt_id, "prompt_id_123") + self.assertEqual(full_response.prompt_state, 2) + self.assertTrue(full_response.is_announcement) + self.assertEqual(full_response.time_created, current_time) + self.assertEqual(full_response.source, "test_source") + + # Test alternate field names (validate_inputs validator) + alternate_kwargs = { + "nick": "submind2", + "shout": "Message using alternate field names", + "responded_shout": "parent_message_id", + "time": current_time.timestamp(), + "message_id": "test_message_id", + "conversation_state": 2, + } + + alternate_response = ChatbotsMqSubmindResponse(**alternate_kwargs) + self.assertEqual(alternate_response.user_id, "submind2") + self.assertEqual(alternate_response.message_text, "Message using alternate field names") + self.assertEqual(alternate_response.replied_message, "parent_message_id") + + # Test model_dump serialization + serialized = full_response.model_dump(by_alias=True) + self.assertEqual(serialized["userID"], "submind1") + self.assertEqual(serialized["messageText"], "Full test response") + self.assertTrue(serialized["isAnnouncement"]) + self.assertEqual(serialized["nick"], "submind1") # Check backwards compatibility field + self.assertEqual(serialized["shout"], "Full test response") # Check backwards compatibility field + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqSubmindResponse(messageText="Missing user_id", message_id="test_id") + + with self.assertRaises(ValidationError): + ChatbotsMqSubmindResponse(userID="test_user", message_id="test_id") + + def test_chatbots_mq_response_type_adapter(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqResponse + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindResponse, ChatbotsMqSavePrompt, ChatbotsMqNewPrompt + from neon_data_models.enum import CcaiControl + from datetime import datetime, timezone + + current_time = datetime.now(tz=timezone.utc) + + # Test regular submind response + regular_message = { + "userID": "submind1", + "messageText": "Regular response", + "conversation_state": 2, + "message_id": "test_message_id" + } + + result = ChatbotsMqResponse(**regular_message) + self.assertIsInstance(result, ChatbotsMqSubmindResponse) + self.assertEqual(result.user_id, "submind1") + self.assertEqual(result.message_text, "Regular response") + + # Test new prompt message + new_prompt_message = { + "routing_key": None, + "message_id": "8ee915f080ae416da1f31081cc4f945a", + "sid": "", + "cid": "35d5dff220", + "title": "", + "user_id": "proctor-f3277918bcc947359994f711452450b6", + "username": None, + "message_text": "!MSG:CREATE_PROMPT", + "replied_message": None, + "bot": "1", + "prompt_id": "f747c1f3caeb4975983b2eb52c35491e", + "prompt_text": "Why is testing important?", + "prompt_state": 1, + "is_announcement": False, + "time_created": current_time, + "source": "klat_observer", + "bot_type": "proctor", + "service_name": "proctor", + "context": {}, + "dom": "", + "omit_reply": True, + "no_save": False, + "created_on": 1743531762, + } + + alt_new_prompt_message = { + "nick": "proctor-ac18f03d0937490080c798d3b242ecd0", + "bot_type": "proctor", + "service_name": "proctor", + "cid": "35d5dff220", + "dom": "", + "conversation_state": 1, + "responded_shout": None, + "shout": "!MSG:CREATE_PROMPT", + "context": {}, + "prompt_id": "00cbbd4c9f33422f9628c7b0c2e90c5b", + "time": "1743539074", + "prompt_text": "Why is testing important?", + "created_on": 1743539074, + "omit_reply": True, + "no_save": False, + "message_id": "f7affd3166244e74ba814f92ccd26eb7", + "bot": "1" + } + + result = ChatbotsMqResponse(**new_prompt_message) + self.assertIsInstance(result, ChatbotsMqNewPrompt) + self.assertEqual(result.prompt_id, new_prompt_message['prompt_id']) + + alt_response = ChatbotsMqResponse(**alt_new_prompt_message) + self.assertIsInstance(alt_response, ChatbotsMqNewPrompt) + self.assertEqual(alt_response.user_id, alt_new_prompt_message['nick']) + + + # Test save prompt message + save_prompt_message = { + "routing_key": None, + "message_id": "e2ad6e92438f4ec0acaa386e22e24954", + "sid": "", + "cid": "35d5dff220", + "title": "", + "user_id": "proctor-f3277918bcc947359994f711452450b6", + "username": None, + "message_text": "!MSG:SAVE_PROMPT_RESULTS", + "replied_message": None, + "bot": "1", + "prompt_id": "prompt_123", + "prompt_state": 4, + "is_announcement": False, + "time_created": current_time, + "source": "klat_observer", + "bot_type": "proctor", + "service_name": "proctor", + "context": { + "winner": "submind1", + "prompt_text": "Test prompt" + }, + "dom": "", + "omit_reply": True, + "no_save": False + } + alt_save_prompt_message = { + "nick": "proctor-ac18f03d0937490080c798d3b242ecd0", + "bot_type": "proctor", + "service_name": "proctor", + "cid": "35d5dff220", + "dom": "", + "conversation_state": 4, + "responded_shout": None, + "shout": "!MSG:SAVE_PROMPT_RESULTS", + "context": {}, + "prompt_id": "", + "time": "1743539090", + "omit_reply": True, + "conversation_context": { + "message_id": "66d5c444961243d9bf95c5a068b0211a", + "state": 4, + "prompt": { + "routing_key": None, + "message_id": "98add3204f", + "sid": "7f80ae0d-a5a9-440d-f113-df167b70be9e", + "cid": "35d5dff220", + "title": "", + "username": "ca45d1ea45134523af7f", + "message_text": "Why is testing important?", + "from_bot": False, + "prompt_id": "00cbbd4c9f33422f9628c7b0c2e90c5b", + "prompt_state": None, + "time_created": 1743539036.0, + "requested_participants": [ + "proctor" + ], + "recipient": None, + "bound_service": "", + "bot": "0", + "messageText": "Why is testing important?", + "nick": "ca45d1ea45134523af7f" + }, + "is_active": True, + "prompt_text": "Why is testing important?", + "available_subminds": [ + "nucleotidings_vllm", + "logistics_vllm", + "neon_vllm" + ], + "nick_mapping": {}, + "participating_subminds": [ + "nucleotidings_vllm", + "logistics_vllm", + "neon_vllm" + ], + "proposed_responses": { + "neon_vllm": "Testing is crucial.", + "logistics_vllm": "Testing is important.", + "nucleotidings_vllm": "Testing is important." + }, + "submind_opinions": { + "neon_vllm": "The answer provided by \"logistics_vllm\" is the best answer", + "nucleotidings_vllm": "The answer provided by 'logistics_vllm' is considered the best.", + "logistics_vllm": "The answer provided by \"nucleotidings_vllm\" is the best answer." + }, + "votes": { + "neon_vllm": "logistics_vllm", + "logistics_vllm": "nucleotidings_vllm", + "nucleotidings_vllm": "logistics_vllm" + }, + "votes_per_submind": { + "logistics_vllm": [ + "neon_vllm", + "nucleotidings_vllm" + ], + "nucleotidings_vllm": [ + "logistics_vllm" + ] + }, + "winner": "logistics_vllm" + }, + "no_save": False, + "message_id": "6924c0422b9347ae9604e4e97cd0847a", + "bot": "1" + } + + result = ChatbotsMqResponse(**save_prompt_message) + self.assertIsInstance(result, ChatbotsMqSavePrompt) + self.assertEqual(result.prompt_id, "prompt_123") + # self.assertIsNone(result.created_on) + self.assertEqual(result.context.winner, "submind1") + self.assertIsInstance(ChatbotsMqResponse(**alt_save_prompt_message), + ChatbotsMqSavePrompt) + + # Test with message_text vs messageText + alternate_syntax = { + "userID": "submind1", + "message_text": "Using message_text instead of messageText", + "conversation_state": 2, + "message_id": "test_message_id" + } + + result = ChatbotsMqResponse(**alternate_syntax) + self.assertIsInstance(result, ChatbotsMqSubmindResponse) + self.assertEqual(result.message_text, "Using message_text instead of messageText") + + def test_chatbots_mq_submind_connection(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindConnection + from datetime import datetime, timezone + + # Test basic initialization + current_time = datetime.now(tz=timezone.utc) + valid_kwargs = { + "nick": "submind1", + "time": current_time, + "message_id": "test_message_id" + } + + connection = ChatbotsMqSubmindConnection(**valid_kwargs) + self.assertIsInstance(connection, ChatbotsMqSubmindConnection) + self.assertEqual(connection.user_id, "submind1") # Check aliased field + self.assertEqual(connection.time, current_time) + + # Test with all fields + minimal_kwargs = { + "user_id": "submind2", + "time": current_time, + "cids": ["conversation1", "conversation2"], + "supports_raw_conversation": True, + "message_id": "test_message_id" + } + + full_connection = ChatbotsMqSubmindConnection(**minimal_kwargs) + self.assertEqual(full_connection.user_id, "submind2") + self.assertEqual(full_connection.cids, ["conversation1", "conversation2"]) + + # With context + minimal_kwargs = { + "user_id": "submind_2-longuuidstring", + "time": current_time, + "cids": ["conversation1", "conversation2"], + "supports_raw_conversation": True, + "message_id": "test_message_id", + "context": {} + } + parsed_service_name = ChatbotsMqSubmindConnection(**minimal_kwargs) + self.assertEqual(parsed_service_name.context.service_name, "submind_2") + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqSubmindConnection(time=current_time, message_id="test_id") + + with self.assertRaises(ValidationError): + ChatbotsMqSubmindConnection(user_id="submind1", time=current_time, + cids="invalid_type") + + def test_chatbots_mq_submind_disconnection(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindDisconnection + + # Test basic initialization + valid_kwargs = { + "nick": "submind1", + "message_id": "test_message_id" + } + + disconnection = ChatbotsMqSubmindDisconnection(**valid_kwargs) + self.assertIsInstance(disconnection, ChatbotsMqSubmindDisconnection) + self.assertEqual(disconnection.user_id, "submind1") # Check aliased field + + # Test with user_id instead of nick + alternate_kwargs = { + "user_id": "submind2", + "message_id": "test_message_id" + } + + alternate_disconnection = ChatbotsMqSubmindDisconnection(**alternate_kwargs) + self.assertEqual(alternate_disconnection.user_id, "submind2") + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqSubmindDisconnection(message_id="test_id") + + def test_chatbots_mq_submind_invitation(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindInvitation + + # Test basic initialization + valid_kwargs = { + "cid": "conversation123", + "requested_participants": ["submind1", "submind2"], + "message_id": "test_message_id" + } + + invitation = ChatbotsMqSubmindInvitation(**valid_kwargs) + self.assertIsInstance(invitation, ChatbotsMqSubmindInvitation) + self.assertEqual(invitation.cid, "conversation123") + self.assertEqual(invitation.requested_participants, ["submind1", "submind2"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqSubmindInvitation(cid="conversation123", message_id="test_id") + + with self.assertRaises(ValidationError): + ChatbotsMqSubmindInvitation(requested_participants=["submind1"], message_id="test_id") + + def test_chatbots_mq_update_participating_subminds(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqUpdateParticipatingSubminds + + # Test basic initialization with required fields + valid_kwargs = { + "cid": "conversation123", + "message_id": "test_message_id" + } + + update = ChatbotsMqUpdateParticipatingSubminds(**valid_kwargs) + self.assertIsInstance(update, ChatbotsMqUpdateParticipatingSubminds) + self.assertEqual(update.cid, "conversation123") + self.assertEqual(update.subminds_to_invite, []) # Default value + self.assertEqual(update.subminds_to_kick, []) # Default value + + # Test with all fields + full_kwargs = { + "cid": "conversation123", + "subminds_to_invite": ["new_submind1", "new_submind2"], + "subminds_to_kick": ["old_submind1"], + "message_id": "test_message_id" + } + + full_update = ChatbotsMqUpdateParticipatingSubminds(**full_kwargs) + self.assertEqual(full_update.subminds_to_invite, ["new_submind1", "new_submind2"]) + self.assertEqual(full_update.subminds_to_kick, ["old_submind1"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqUpdateParticipatingSubminds(subminds_to_invite=["submind1"], message_id="test_id") + + # TODO: Should this raise an exception if the request does nothing? + # with self.assertRaises(ValidationError): + # ChatbotsMqUpdateParticipatingSubminds(cid="conversation123") + + def test_chatbots_mq_submind_conversation_ban(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindConversationBan + + # Test basic initialization + valid_kwargs = { + "nick": "submind1", + "cid": "conversation123", + "message_id": "test_message_id" + } + + ban = ChatbotsMqSubmindConversationBan(**valid_kwargs) + self.assertIsInstance(ban, ChatbotsMqSubmindConversationBan) + self.assertEqual(ban.user_id, "submind1") # Check aliased field + self.assertEqual(ban.cid, "conversation123") + + # Test with user_id instead of nick + alternate_kwargs = { + "user_id": "submind2", + "cid": "conversation456", + "message_id": "test_message_id" + } + + alternate_ban = ChatbotsMqSubmindConversationBan(**alternate_kwargs) + self.assertEqual(alternate_ban.user_id, "submind2") + self.assertEqual(alternate_ban.cid, "conversation456") + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqSubmindConversationBan(user_id="submind1", message_id="test_id") + + with self.assertRaises(ValidationError): + ChatbotsMqSubmindConversationBan(cid="conversation123", message_id="test_id") + + def test_chatbots_mq_submind_global_ban(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindGlobalBan + + # Test basic initialization + valid_kwargs = { + "nick": "submind1", + "message_id": "test_message_id" + } + + ban = ChatbotsMqSubmindGlobalBan(**valid_kwargs) + self.assertIsInstance(ban, ChatbotsMqSubmindGlobalBan) + self.assertEqual(ban.user_id, "submind1") # Check aliased field + + # Test with user_id instead of nick + alternate_kwargs = { + "user_id": "submind2", + "message_id": "test_message_id" + } + + alternate_ban = ChatbotsMqSubmindGlobalBan(**alternate_kwargs) + self.assertEqual(alternate_ban.user_id, "submind2") + + # Test missing required fields + with self.assertRaises(ValidationError): + ChatbotsMqSubmindGlobalBan(message_id="test_id") + + def test_chatbots_mq_submind_response_error(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindResponseError + + # Test basic initialization + valid_kwargs = { + "message": "An error occurred", + "message_id": "test_message_id" + } + + error_response = ChatbotsMqSubmindResponseError(**valid_kwargs) + self.assertIsInstance(error_response, ChatbotsMqSubmindResponseError) + self.assertEqual(error_response.message, "An error occurred") + self.assertEqual(error_response.message_id, "test_message_id") + + # Test with alias field + alias_kwargs = { + "msg": "An error with alias field", + "message_id": "test_message_id" + } + + alias_response = ChatbotsMqSubmindResponseError(**alias_kwargs) + self.assertEqual(alias_response.message, "An error with alias field") + + # Test serialization + serialized = error_response.model_dump() + self.assertEqual(serialized["message"], "An error occurred") + + # Test by_alias serialization + serialized_alias = error_response.model_dump(by_alias=True) + self.assertEqual(serialized_alias["msg"], "An error occurred") + + # Test without message (should be None by default) + no_message = ChatbotsMqSubmindResponseError(message_id="test_message_id") + self.assertIsNone(no_message.message) + + def test_chatbots_mq_response_edge_cases(self): + from neon_data_models.models.api.mq.chatbots import ChatbotsMqResponse + from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindResponse + + # Test with empty/minimal valid message + minimal_message = { + "userID": "test_user", + "messageText": "", + "message_id": "test_message_id" + } + + result = ChatbotsMqResponse(**minimal_message) + self.assertIsInstance(result, ChatbotsMqSubmindResponse) + self.assertEqual(result.message_text, "") + + # Test with mixed aliased and non-aliased fields + mixed_fields = { + "userID": "test_user", + "shout": "Using aliased shout field", + "message_id": "test_message_id", + "nick": "nick_value" # This should be used for user_id too + } + + result = ChatbotsMqResponse(**mixed_fields) + self.assertIsInstance(result, ChatbotsMqSubmindResponse) + self.assertEqual(result.message_text, "Using aliased shout field") + self.assertEqual(result.user_id, "test_user") # userID takes precedence over nick + + # Test with malformed timestamps that should be converted + timestamp_message = { + "userID": "test_user", + "messageText": "Timestamp test", + "created_on": "1611234567", # String timestamp + "time": 1611234567, # Integer timestamp + "message_id": "test_message_id" + } + + result = ChatbotsMqResponse(**timestamp_message) + self.assertIsInstance(result, ChatbotsMqSubmindResponse) + # Ensure the timestamps are properly handled in serialization + serialized = result.model_dump(by_alias=True) + self.assertEqual(serialized["created_on"], 1611234567) + + # Test with null/None fields that should be handled + null_fields = { + "userID": "test_user", + "messageText": "Test with nulls", + "message_id": "test_message_id", + "sid": None, # Should be handled by the model validator + "repliedMessage": None, + "promptID": None + } + + result = ChatbotsMqResponse(**null_fields) + self.assertIsInstance(result, ChatbotsMqSubmindResponse) + self.assertEqual(result.sid, "") # Default value when None is provided diff --git a/tests/models/test_base.py b/tests/models/test_base.py index 7d5bd337..9bf4d988 100644 --- a/tests/models/test_base.py +++ b/tests/models/test_base.py @@ -365,8 +365,9 @@ def test_klat_context(self): def test_mq_context(self): from neon_data_models.models.base.contexts import MQContext - with self.assertRaises(ValidationError): - MQContext() + default = MQContext() + self.assertIsInstance(default.message_id, str) + self.assertNotEqual(default.message_id, MQContext().message_id) minimal_ctx = MQContext(message_id="test_message_id_string") self.assertEqual(minimal_ctx, MQContext(**minimal_ctx.model_dump())) From 814aa9bece3d593b942517a3b73455fc79141c61 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Fri, 4 Apr 2025 17:03:56 +0000 Subject: [PATCH 04/36] Increment Version to 0.0.2a2 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 47f0f122..f453f6c8 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a1" +__version__ = "0.0.2a2" From 301020f1a33e9bf17c90affb44bce036ad7f875e Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Tue, 8 Apr 2025 13:12:51 -0700 Subject: [PATCH 05/36] Define `gradio` context with `session` key used to track browser sessions (#20) --- neon_data_models/models/api/mq/chatbots.py | 4 ++-- neon_data_models/models/base/contexts.py | 4 ++++ neon_data_models/models/base/messagebus.py | 6 +++++- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/neon_data_models/models/api/mq/chatbots.py b/neon_data_models/models/api/mq/chatbots.py index de81601a..9cf04d8a 100644 --- a/neon_data_models/models/api/mq/chatbots.py +++ b/neon_data_models/models/api/mq/chatbots.py @@ -61,7 +61,7 @@ class ChatbotsMqRequest(KlatContext, MQContext): default=None, description="List of CCAI participants requested to handle the shout") recipient: Optional[str] = Field( - default=None, description="Explicitly defined recipient of the shout. ") + default=None, description="Explicitly defined recipient of the shout") bound_service: Optional[str] = Field( default=None, description="Service bound to the conversation") @@ -82,7 +82,7 @@ def from_sio_message(cls, sio_message: dict) -> 'ChatbotsMqRequest': recipient=sio_message.get("recipient"), bound_service=sio_message.get("bound_service"), ) - + def model_dump(self, **kwargs): """ Override model_dump to include SIO fields for backwards compatibility diff --git a/neon_data_models/models/base/contexts.py b/neon_data_models/models/base/contexts.py index 1a560191..c48e683a 100644 --- a/neon_data_models/models/base/contexts.py +++ b/neon_data_models/models/base/contexts.py @@ -151,3 +151,7 @@ def validate_inputs(cls, values): # Allow default_factory to generate a message_id values.pop("message_id") return values + + +class GradioContext(BaseModel): + session: str = Field(description="Gradio session ID") diff --git a/neon_data_models/models/base/messagebus.py b/neon_data_models/models/base/messagebus.py index 58871b61..f663927d 100644 --- a/neon_data_models/models/base/messagebus.py +++ b/neon_data_models/models/base/messagebus.py @@ -28,7 +28,8 @@ from pydantic import ConfigDict, Field, field_validator, model_validator from neon_data_models.models.base import BaseModel -from neon_data_models.models.base.contexts import (SessionContext, KlatContext, +from neon_data_models.models.base.contexts import (GradioContext, + SessionContext, KlatContext, TimingContext, MQContext) from neon_data_models.models.client.node import NodeData from neon_data_models.models.user.database import NeonUserConfig @@ -61,6 +62,9 @@ class MessageContext(BaseModel): description="List of destination modules expected to handle the message" ) neon_should_respond: bool = True + gradio: Optional[GradioContext] = Field( + default=None, + description="Context for messages originating from a Gradio interface") @field_validator('destination') def validate_destination(cls, v): From bcbb763428ab519c16a62a01102c6634e5378822 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Tue, 8 Apr 2025 20:13:05 +0000 Subject: [PATCH 06/36] Increment Version to 0.0.2a3 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index f453f6c8..ec67a2bb 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a2" +__version__ = "0.0.2a3" From 1a1f709e4d1f0935ef18687db2ecb7b70e1290eb Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Thu, 10 Apr 2025 11:38:17 -0700 Subject: [PATCH 07/36] Implement models for BrainForge service (#8) * Define models for BrainForge services * Allow empty string `system_prompt` in `LLMPersona` to support "vanilla" BrainForge personas * Fix fields defined with descriptions as default values * Add `BrainForgeLLM.vllm_spec` convenience property Add `LLMGetInference.as_llm_request` convenience method * Start defining HTTP models for brainforge endpoints * Add HTTP models for BrainForge service endpoints * Add missing `models.api.http` init file * Append user `query` to history in `LLMRequest.to_completion_kwargs` * Add missing license notice * Add docstrings to clarify submodule intended usage * Update documentation and tests to account for history handling change * Fix test error * Allow persona definition with `None` system prompt * Prevent inserting `None` system prompt in completion request history * Define models for generic completion requests/responses * Define models for tokenizer request/response * Update raw endpoint models based on actual usage * Address review feedback regarding chat template and inference request handling Refactor tokenizer model names to be more descriptive * Update `__all__` to include added module and models * Add MQ module unit test coverage (mostly Claude3.7-generated with some manual adjustments) --- neon_data_models/models/api/http/__init__.py | 31 ++ .../models/api/http/brainforge.py | 57 ++++ neon_data_models/models/api/llm.py | 32 +- neon_data_models/models/api/mq/__init__.py | 5 + neon_data_models/models/api/mq/brainforge.py | 115 +++++++ .../models/api/node_v1/__init__.py | 5 + tests/models/api/test_llm.py | 6 +- tests/models/api/test_mq.py | 303 ++++++++++++++++++ 8 files changed, 545 insertions(+), 9 deletions(-) create mode 100644 neon_data_models/models/api/http/__init__.py create mode 100644 neon_data_models/models/api/http/brainforge.py create mode 100644 neon_data_models/models/api/mq/brainforge.py diff --git a/neon_data_models/models/api/http/__init__.py b/neon_data_models/models/api/http/__init__.py new file mode 100644 index 00000000..19a7ca21 --- /dev/null +++ b/neon_data_models/models/api/http/__init__.py @@ -0,0 +1,31 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2024 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from neon_data_models.models.api.http.brainforge import * + +""" +This module contains models for interacting via HANA (HTTP). +""" diff --git a/neon_data_models/models/api/http/brainforge.py b/neon_data_models/models/api/http/brainforge.py new file mode 100644 index 00000000..7f099e3b --- /dev/null +++ b/neon_data_models/models/api/http/brainforge.py @@ -0,0 +1,57 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2024 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from typing import List, Optional +from pydantic import Field + +from neon_data_models.models.base import BaseModel +from neon_data_models.models.api.llm import BrainForgeLLM, LLMPersona, LLMRequest + + +class LLMGetModelsHttpResponse(BaseModel): + models: List[BrainForgeLLM] + + +class LLMGetPersonasHttpRequest(BaseModel): + model_id: str = Field( + description="Model ID (@) to get personas for") + + +class LLMGetPersonasHttpResponse(BaseModel): + personas: List[LLMPersona] = Field( + description="List of personas associated with the requested model.") + + +class LLMGetInferenceHttpRequest(LLMRequest): + llm_name: str = Field(description="Model name to request") + llm_revision: str = Field(description="Model revision to request") + model: Optional[str] = None + + +__all__ = [LLMGetModelsHttpResponse.__name__, + LLMGetPersonasHttpRequest.__name__, + LLMGetPersonasHttpResponse.__name__, + LLMGetInferenceHttpRequest.__name__] diff --git a/neon_data_models/models/api/llm.py b/neon_data_models/models/api/llm.py index d78119ca..10820125 100644 --- a/neon_data_models/models/api/llm.py +++ b/neon_data_models/models/api/llm.py @@ -57,7 +57,7 @@ class LLMPersona(LLMPersonaIdentity): """ description: Optional[str] = Field( None, description="Human-readable description of this persona") - system_prompt: str = Field( + system_prompt: Optional[str] = Field( None, description="System prompt associated with this persona. " "If None, `description` will be used.") enabled: bool = Field( @@ -71,7 +71,7 @@ def validate_request(self): self.system_prompt = None return self - assert any((self.description, self.system_prompt)) + assert any(x is not None for x in (self.description, self.system_prompt)) if self.system_prompt is None: self.system_prompt = self.description return self @@ -86,7 +86,7 @@ class LLMRequest(BaseModel): "OpenAI-compatible requests.") persona: LLMPersona = Field( description="Requested persona to respond to this message") - model: str = Field(description="Model to request") + model: str = Field(description="Model to request (@)") max_tokens: int = Field( default=512, ge=64, le=2048, description="Maximum number of tokens to include in the response") @@ -110,7 +110,8 @@ class LLMRequest(BaseModel): "Mutually exclusive with `stream`.") max_history: int = Field( default=2, description="Maximum number of user/assistant " - "message pairs to include in history context.") + "message pairs to include in history context. " + "Excludes system prompt and incoming query.") @model_validator(mode='before') @classmethod @@ -180,8 +181,10 @@ def to_completion_kwargs(self, mq2role: dict = None) -> dict: history = self.messages[-2*self.max_history:] for msg in history: msg["role"] = mq2role.get(msg["role"]) or msg["role"] - history.insert(0, {"role": "system", - "content": self.persona.system_prompt}) + if self.persona.system_prompt is not None: + history.insert(0, {"role": "system", + "content": self.persona.system_prompt}) + history.append({"role": "user", "content": self.query}) return {"model": self.model, "messages": history, "max_tokens": self.max_tokens, @@ -212,5 +215,20 @@ def validate_inputs(cls, values): return values +class BrainForgeLLM(BaseModel): + name: str = Field(description="LLM Name") + version: str = Field(description="LLM Version") + personas: List[LLMPersona] = Field( + default=[], description="List of personas defined in this model") + + @property + def vllm_spec(self): + """ + Model identifier used by vllm (@) + """ + return f"{self.name}@{self.version}" + + __all__ = [LLMPersonaIdentity.__name__, LLMPersona.__name__, - LLMRequest.__name__, LLMResponse.__name__] + LLMRequest.__name__, LLMResponse.__name__, + BrainForgeLLM.__name__] diff --git a/neon_data_models/models/api/mq/__init__.py b/neon_data_models/models/api/mq/__init__.py index da32c610..219bf706 100644 --- a/neon_data_models/models/api/mq/__init__.py +++ b/neon_data_models/models/api/mq/__init__.py @@ -24,7 +24,12 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +from neon_data_models.models.api.mq.brainforge import * from neon_data_models.models.api.mq.chatbots import * from neon_data_models.models.api.mq.llm import * from neon_data_models.models.api.mq.neon import * from neon_data_models.models.api.mq.users import * + +""" +This module contains models for interacting via the MQ bus. +""" diff --git a/neon_data_models/models/api/mq/brainforge.py b/neon_data_models/models/api/mq/brainforge.py new file mode 100644 index 00000000..5f484a7f --- /dev/null +++ b/neon_data_models/models/api/mq/brainforge.py @@ -0,0 +1,115 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2024 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from typing import List, Optional, Any, Dict, Literal, Union +from pydantic import Field + +from neon_data_models.models.base.contexts import MQContext +from neon_data_models.models.api.llm import BrainForgeLLM, LLMRequest, LLMResponse, LLMPersona +from neon_data_models.models.api.http.brainforge import LLMGetModelsHttpResponse, LLMGetPersonasHttpRequest + + +class LLMGetModels(MQContext): + user_id: str = Field( + description="ID of user to get models for") + + +class LLMGetModelsResponse(MQContext, LLMGetModelsHttpResponse): + pass + + +class LLMGetPersonas(LLMGetModels, LLMGetPersonasHttpRequest): + @property + def model_name(self): + return self.model_id.split("@")[0] + + @property + def model_version(self): + return self.model_id.split("@", 1)[1] + + +class LLMGetPersonasResponse(MQContext): + model: Optional[BrainForgeLLM] = Field( + "Full configuration of requested model if model is loaded and access " + "is allowed, else None.") + + @property + def personas(self) -> List[LLMPersona]: + """ + Convenience property defined to easily reference the personas requested + """ + return self.model.personas if self.model else [] + + +class LLMGetInference(LLMRequest, MQContext): + user_id: str = Field("ID of user making the request") + + def as_llm_request(self): + """ + Get a plain `LLMRequest` object from this `LLMGetInference` object. + """ + return LLMRequest(**self.model_dump()) + + +class LLMGetCompletion(LLMGetModels): + model: str = Field(description="Model to request (@)") + completion_kwargs: Dict[str, Any] = Field( + description="Dictionary of kwargs to pass to OpenAI completion request") + + +class LLMGetCompletionResponse(MQContext): + openai_response: dict = Field( + description="OpenAI ChatCompletion model") + + +class LLMGetTokenizerChatTemplatedString(LLMGetModels): + model: str = Field(description="Model to request (@)") + messages: List[Dict[Literal["role", "content"], str]] = Field( + description="List of dict messages in OpenAI format") + add_generation_prompt: bool = Field( + description="If true, assistant start tokens will be appended to the " + "formatted output.") + tokenize: bool = Field( + False, + description="If true, a list of token strings is returned, " + "else a single string") + + +class LLMGetTokenizerChatTemplatedStringResponse(MQContext): + prompt: Union[List[str], str] = Field( + description="Prompt generated by the tokenizer") + + +class LLMGetInferenceResponse(LLMResponse, MQContext): + pass + + +__all__ = [LLMGetModels.__name__, LLMGetModelsResponse.__name__, + LLMGetPersonas.__name__, LLMGetPersonasResponse.__name__, + LLMGetInference.__name__, LLMGetInferenceResponse.__name__, + LLMGetCompletion.__name__, LLMGetCompletionResponse.__name__, + LLMGetTokenizerChatTemplatedString.__name__, + LLMGetTokenizerChatTemplatedStringResponse.__name__] diff --git a/neon_data_models/models/api/node_v1/__init__.py b/neon_data_models/models/api/node_v1/__init__.py index 54700848..71e9bcd8 100644 --- a/neon_data_models/models/api/node_v1/__init__.py +++ b/neon_data_models/models/api/node_v1/__init__.py @@ -33,6 +33,11 @@ from neon_data_models.models.base.messagebus import BaseMessage, MessageContext +""" +This module contains models for interacting via the Node socket (WS). +""" + + class AudioInputData(BaseModel): audio_data: str = Field(description="base64-encoded audio") lang: str = Field(description="BCP-47 language code") diff --git a/tests/models/api/test_llm.py b/tests/models/api/test_llm.py index 3ae9b0ee..153be3cc 100644 --- a/tests/models/api/test_llm.py +++ b/tests/models/api/test_llm.py @@ -60,7 +60,7 @@ def test_llm_persona(self): # Under-defined persona with self.assertRaises(ValidationError): - LLMPersona(name="underdefined persona") + LLMPersona(description="underdefined persona") # Valid vanilla persona vanilla = LLMPersona(name="vanilla") @@ -91,7 +91,9 @@ def test_llm_request(self): self.assertFalse(valid_request.beam_search) self.assertEqual(len(valid_request.history), len(test_history)) self.assertEqual(len(valid_request.to_completion_kwargs()['messages']), - 2 * valid_request.max_history + 1) + 2 * valid_request.max_history + 2) + self.assertEqual(valid_request.to_completion_kwargs()['messages'][-1]['content'], + test_query) # Valid explicit streaming streaming_request = LLMRequest(query=test_query, history=test_history, diff --git a/tests/models/api/test_mq.py b/tests/models/api/test_mq.py index 14a62038..d1084387 100644 --- a/tests/models/api/test_mq.py +++ b/tests/models/api/test_mq.py @@ -2105,3 +2105,306 @@ def test_chatbots_mq_response_edge_cases(self): result = ChatbotsMqResponse(**null_fields) self.assertIsInstance(result, ChatbotsMqSubmindResponse) self.assertEqual(result.sid, "") # Default value when None is provided + + +class TestBrainforgeMQ(TestCase): + def test_llm_get_models(self): + from neon_data_models.models.api.mq.brainforge import LLMGetModels + from neon_data_models.models.base.contexts import MQContext + + # Test valid initialization + valid_kwargs = {"user_id": "test_user", "message_id": "test_id"} + models_request = LLMGetModels(**valid_kwargs) + self.assertIsInstance(models_request, LLMGetModels) + self.assertIsInstance(models_request, MQContext) + self.assertEqual(models_request.user_id, "test_user") + + # Test missing required fields + with self.assertRaises(ValidationError): + LLMGetModels(message_id="test_id") # Missing user_id + + def test_llm_get_models_response(self): + from neon_data_models.models.api.mq.brainforge import LLMGetModelsResponse + from neon_data_models.models.base.contexts import MQContext + from neon_data_models.models.api.http.brainforge import LLMGetModelsHttpResponse + + # Test valid initialization + valid_kwargs = { + "models": [{"version": "v1", "name": "model_1"}], + "message_id": "test_id" + } + models_response = LLMGetModelsResponse(**valid_kwargs) + self.assertIsInstance(models_response, LLMGetModelsResponse) + self.assertIsInstance(models_response, LLMGetModelsHttpResponse) + self.assertIsInstance(models_response, MQContext) + self.assertEqual(len(models_response.models), 1) + self.assertEqual(models_response.models[0].vllm_spec, "model_1@v1") + + # Test missing required fields + with self.assertRaises(ValidationError): + LLMGetModelsResponse(message_id="test_id") # Missing models + + def test_llm_get_personas(self): + from neon_data_models.models.api.mq.brainforge import LLMGetPersonas + from neon_data_models.models.api.mq.brainforge import LLMGetModels + from neon_data_models.models.api.http.brainforge import LLMGetPersonasHttpRequest + + # Test valid initialization + valid_kwargs = { + "user_id": "test_user", + "model_id": "model1@v1.0", + "message_id": "test_id" + } + personas_request = LLMGetPersonas(**valid_kwargs) + self.assertIsInstance(personas_request, LLMGetPersonas) + self.assertIsInstance(personas_request, LLMGetModels) + self.assertIsInstance(personas_request, LLMGetPersonasHttpRequest) + + # Test properties + self.assertEqual(personas_request.model_name, "model1") + self.assertEqual(personas_request.model_version, "v1.0") + + # Test with complex model version + complex_version_kwargs = { + "user_id": "test_user", + "model_id": "model1@version/with/slashes", + "message_id": "test_id" + } + complex_request = LLMGetPersonas(**complex_version_kwargs) + self.assertEqual(complex_request.model_name, "model1") + self.assertEqual(complex_request.model_version, "version/with/slashes") + + # Test missing required fields + with self.assertRaises(ValidationError): + LLMGetPersonas(user_id="test_user", message_id="test_id") # Missing model_id + + with self.assertRaises(ValidationError): + LLMGetPersonas(model_id="model1@v1", message_id="test_id") # Missing user_id + + def test_llm_get_personas_response(self): + from neon_data_models.models.api.mq.brainforge import LLMGetPersonasResponse + from neon_data_models.models.base.contexts import MQContext + from neon_data_models.models.api.llm import BrainForgeLLM, LLMPersona + + # Test with model (personas available) + model_with_personas = BrainForgeLLM( + version="v1", + name="model1", + personas=[ + LLMPersona(name="persona1", system_prompt="System prompt 1"), + LLMPersona(name="persona2", system_prompt="System prompt 2") + ] + ) + + valid_kwargs = { + "model": model_with_personas, + "message_id": "test_id" + } + + personas_response = LLMGetPersonasResponse(**valid_kwargs) + self.assertIsInstance(personas_response, LLMGetPersonasResponse) + self.assertIsInstance(personas_response, MQContext) + + # Test personas property + self.assertEqual(len(personas_response.personas), 2) + self.assertEqual(personas_response.personas[0].name, "persona1") + self.assertEqual(personas_response.personas[1].name, "persona2") + + # Test with model=None + no_model_kwargs = {"model": None, "message_id": "test_id"} + no_model_response = LLMGetPersonasResponse(**no_model_kwargs) + self.assertEqual(no_model_response.personas, []) + + # # Test missing required field (message_id) + # with self.assertRaises(ValidationError): + # LLMGetPersonasResponse(model=model_with_personas) + + def test_llm_get_inference(self): + from neon_data_models.models.api.mq.brainforge import LLMGetInference + from neon_data_models.models.api.llm import LLMRequest + + # Test with all optional parameters + full_kwargs = { + "user_id": "test_user", + "query": "test query", + "history": [("user", "previous message")], + "model": "test_model", + "persona": {"name": "test_persona", "system_prompt": "test prompt"}, + "message_id": "test_id", + "context": {"some_key": "some_value"} + } + full_request = LLMGetInference(**full_kwargs) + self.assertEqual(full_request.user_id, "test_user") + self.assertEqual(full_request.model, "test_model") + + # Test as_llm_request method + llm_request = full_request.as_llm_request() + self.assertIsInstance(llm_request, LLMRequest) + self.assertNotIsInstance(llm_request, LLMGetInference) # Should not be a LLMGetInference + self.assertEqual(llm_request.query, "test query") + self.assertEqual(llm_request.model, "test_model") + + # Test missing required fields + with self.assertRaises(ValidationError): + LLMGetInference(user_id="test_user", history=[], message_id="test_id") # Missing query + + with self.assertRaises(ValidationError): + LLMGetInference(query="test query", history=[], message_id="test_id") # Missing user_id + + def test_llm_get_inference_response(self): + from neon_data_models.models.api.mq.brainforge import LLMGetInferenceResponse + from neon_data_models.models.base.contexts import MQContext + from neon_data_models.models.api.llm import LLMResponse + + # Test valid initialization + valid_kwargs = { + "response": "This is a test response", + "message_id": "test_id", + "history": [("user", "prompt"), ("assistant", "response")], + } + inference_response = LLMGetInferenceResponse(**valid_kwargs) + self.assertIsInstance(inference_response, LLMGetInferenceResponse) + self.assertIsInstance(inference_response, LLMResponse) + self.assertIsInstance(inference_response, MQContext) + self.assertEqual(inference_response.response, "This is a test response") + + # Test with all optional fields + full_kwargs = { + "response": "Test response with metadata", + "model": "test_model", + "persona": {"name": "test_persona"}, + "message_id": "test_id", + "context": {"some_key": "some_value"}, + "history": [("user", "prompt"), ("assistant", "response")] + } + full_response = LLMGetInferenceResponse(**full_kwargs) + self.assertEqual(full_response.response, full_kwargs["response"]) + # self.assertEqual(full_response.persona["name"], "test_persona") + + # Test missing required fields + with self.assertRaises(ValidationError): + LLMGetInferenceResponse(message_id="test_id") # Missing response + + with self.assertRaises(ValidationError): + LLMGetInferenceResponse(response="test response") # Missing message_id + + def test_llm_get_completion(self): + from neon_data_models.models.api.mq.brainforge import LLMGetCompletion + from neon_data_models.models.api.mq.brainforge import LLMGetModels + + # Test valid initialization with minimum required parameters + valid_kwargs = { + "user_id": "test_user", + "model": "model1@v1", + "completion_kwargs": {"temperature": 0.7, "max_tokens": 100}, + "message_id": "test_id" + } + completion_request = LLMGetCompletion(**valid_kwargs) + self.assertIsInstance(completion_request, LLMGetCompletion) + self.assertIsInstance(completion_request, LLMGetModels) + self.assertEqual(completion_request.model, "model1@v1") + self.assertEqual(completion_request.completion_kwargs["temperature"], 0.7) + + # Test missing required fields + with self.assertRaises(ValidationError): + LLMGetCompletion(user_id="test_user", completion_kwargs={}, message_id="test_id") # Missing model + + with self.assertRaises(ValidationError): + LLMGetCompletion(user_id="test_user", model="model1@v1", message_id="test_id") # Missing completion_kwargs + + def test_llm_get_completion_response(self): + from neon_data_models.models.api.mq.brainforge import LLMGetCompletionResponse + from neon_data_models.models.base.contexts import MQContext + + # Test valid initialization + valid_kwargs = { + "openai_response": { + "id": "chatcmpl-123", + "choices": [{ + "index": 0, + "message": {"role": "assistant", "content": "Test response"}, + "finish_reason": "stop" + }] + }, + "message_id": "test_id" + } + completion_response = LLMGetCompletionResponse(**valid_kwargs) + self.assertIsInstance(completion_response, LLMGetCompletionResponse) + self.assertIsInstance(completion_response, MQContext) + self.assertEqual(completion_response.openai_response["id"], "chatcmpl-123") + + # Test missing required fields + with self.assertRaises(ValidationError): + LLMGetCompletionResponse(message_id="test_id") # Missing openai_response + + def test_llm_get_tokenizer_chat_templated_string(self): + from neon_data_models.models.api.mq.brainforge import LLMGetTokenizerChatTemplatedString + from neon_data_models.models.api.mq.brainforge import LLMGetModels + + # Test valid initialization with minimum required parameters + valid_kwargs = { + "user_id": "test_user", + "model": "model1@v1", + "messages": [ + {"role": "system", "content": "You are a helpful assistant"}, + {"role": "user", "content": "Hello"} + ], + "add_generation_prompt": True, + "message_id": "test_id" + } + tokenizer_request = LLMGetTokenizerChatTemplatedString(**valid_kwargs) + self.assertIsInstance(tokenizer_request, LLMGetTokenizerChatTemplatedString) + self.assertIsInstance(tokenizer_request, LLMGetModels) + self.assertEqual(tokenizer_request.model, "model1@v1") + self.assertEqual(len(tokenizer_request.messages), 2) + self.assertTrue(tokenizer_request.add_generation_prompt) + self.assertFalse(tokenizer_request.tokenize) # Default value + + # Test with tokenize=True + tokenize_kwargs = valid_kwargs.copy() + tokenize_kwargs["tokenize"] = True + tokenize_request = LLMGetTokenizerChatTemplatedString(**tokenize_kwargs) + self.assertTrue(tokenize_request.tokenize) + + # Test missing required fields + with self.assertRaises(ValidationError): + LLMGetTokenizerChatTemplatedString( + user_id="test_user", + messages=[{"role": "user", "content": "Hello"}], + add_generation_prompt=True, + message_id="test_id" + ) # Missing model + + with self.assertRaises(ValidationError): + LLMGetTokenizerChatTemplatedString( + user_id="test_user", + model="model1@v1", + add_generation_prompt=True, + message_id="test_id" + ) # Missing messages + + def test_llm_get_tokenizer_chat_templated_string_response(self): + from neon_data_models.models.api.mq.brainforge import LLMGetTokenizerChatTemplatedStringResponse + from neon_data_models.models.base.contexts import MQContext + + # Test valid initialization with string prompt + string_kwargs = { + "prompt": "This is a templated prompt string", + "message_id": "test_id" + } + string_response = LLMGetTokenizerChatTemplatedStringResponse(**string_kwargs) + self.assertIsInstance(string_response, LLMGetTokenizerChatTemplatedStringResponse) + self.assertIsInstance(string_response, MQContext) + self.assertEqual(string_response.prompt, "This is a templated prompt string") + + # Test with list of tokens + tokens_kwargs = { + "prompt": ["This", "is", "a", "tokenized", "prompt"], + "message_id": "test_id" + } + tokens_response = LLMGetTokenizerChatTemplatedStringResponse(**tokens_kwargs) + self.assertEqual(tokens_response.prompt, ["This", "is", "a", "tokenized", "prompt"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + LLMGetTokenizerChatTemplatedStringResponse(message_id="test_id") # Missing prompt From 0c842ecee834d628c296402eaa1de044487dcd77 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Thu, 10 Apr 2025 18:38:34 +0000 Subject: [PATCH 08/36] Increment Version to 0.0.2a4 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index ec67a2bb..778b8ba3 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a3" +__version__ = "0.0.2a4" From 08b5a82356f026da720b794856b9ee32b0f615eb Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Fri, 18 Apr 2025 18:18:30 -0700 Subject: [PATCH 09/36] Fix type annotation for `user_profiles` context to match expected format (#22) * Fix type annotation for `user_profiles` context to match expected format * Refactor to ensure `UserProfile` is used in Message contexts --- neon_data_models/models/api/mq/neon.py | 4 ++-- neon_data_models/models/base/messagebus.py | 4 ++-- tests/models/test_base.py | 8 ++++---- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/neon_data_models/models/api/mq/neon.py b/neon_data_models/models/api/mq/neon.py index c2881bfc..e22242ac 100644 --- a/neon_data_models/models/api/mq/neon.py +++ b/neon_data_models/models/api/mq/neon.py @@ -31,7 +31,7 @@ from neon_data_models.models.base.contexts import KlatContext, MQContext, \ SessionContext, TimingContext from neon_data_models.models.base.messagebus import BaseMessage, MessageContext -from neon_data_models.models.user.database import NeonUserConfig +from neon_data_models.models.user.neon_profile import UserProfile from neon_data_models.models.api.messagebus import NeonGetLanguages, NeonGetTts, NeonGetStt, \ NeonAudioInput, NeonLanguagesResponse, NeonTextInput, NeonSttResponse, NeonTtsResponse @@ -170,7 +170,7 @@ def from_sio_message(sio_message: dict) -> BaseMessage: client=sio_message.get("client", "unknown"), username=sio_message.get("nick", "guest"), klat_data=klat_context, mq=mq_context, - user_profiles=[NeonUserConfig()], + user_profiles=[UserProfile()], session=SessionContext( session_id=sio_message.get("cid", "klat")), timing=TimingContext( diff --git a/neon_data_models/models/base/messagebus.py b/neon_data_models/models/base/messagebus.py index f663927d..a26a80d4 100644 --- a/neon_data_models/models/base/messagebus.py +++ b/neon_data_models/models/base/messagebus.py @@ -32,7 +32,7 @@ SessionContext, KlatContext, TimingContext, MQContext) from neon_data_models.models.client.node import NodeData -from neon_data_models.models.user.database import NeonUserConfig +from neon_data_models.models.user.neon_profile import UserProfile class MessageContext(BaseModel): @@ -43,7 +43,7 @@ class MessageContext(BaseModel): timing: TimingContext = Field( description="User Interaction Timing Information", default=TimingContext()) - user_profiles: Optional[List[NeonUserConfig]] = ( + user_profiles: Optional[List[UserProfile]] = ( Field(description="List of relevant user profiles", default=None)) klat_data: Optional[KlatContext] = Field( description="Klat context for Klat-generated messages", default=None) diff --git a/tests/models/test_base.py b/tests/models/test_base.py index 9bf4d988..9f8e8fae 100644 --- a/tests/models/test_base.py +++ b/tests/models/test_base.py @@ -378,7 +378,7 @@ def test_base_model(self): from ovos_bus_client.message import Message from neon_data_models.models.base.messagebus import BaseMessage from neon_data_models.models.client import NodeData - from neon_data_models.models.user import NeonUserConfig + from neon_data_models.models.user import UserProfile with self.assertRaises(ValidationError): BaseMessage() @@ -395,7 +395,7 @@ def test_base_model(self): "extra_key": "text"}) # Defined keys will generate objects self.assertIsInstance(message.context.node_data, NodeData) - self.assertIsInstance(message.context.user_profiles[0], NeonUserConfig) + self.assertIsInstance(message.context.user_profiles[0], UserProfile) as_messagebus = message.as_messagebus_message() self.assertIsInstance(as_messagebus, Message) @@ -413,7 +413,7 @@ def test_base_model(self): def test_message_context(self): from neon_data_models.models.base.messagebus import MessageContext from neon_data_models.models.client import NodeData - from neon_data_models.models.user import NeonUserConfig + from neon_data_models.models.user import UserProfile # Default Behavior default_context = MessageContext() @@ -431,7 +431,7 @@ def test_message_context(self): from neon_data_models.models.base.contexts import SessionContext self.assertIsInstance(extra_context.session, SessionContext) self.assertIsInstance(extra_context.node_data, NodeData) - self.assertIsInstance(extra_context.user_profiles[0], NeonUserConfig) + self.assertIsInstance(extra_context.user_profiles[0], UserProfile) from neon_data_models.models.base.contexts import KlatContext self.assertIsInstance(extra_context.klat_data, KlatContext) from neon_data_models.models.base.contexts import MQContext From 7fd09f4f099ae27786fa1d16d8e917d6a1d933c2 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Sat, 19 Apr 2025 01:18:42 +0000 Subject: [PATCH 10/36] Increment Version to 0.0.2a5 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 778b8ba3..5ca8c54b 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a4" +__version__ = "0.0.2a5" From 45f305b8f3d6b21bf8e57b64bb59f4fccf13e16d Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Tue, 27 May 2025 15:46:36 -0700 Subject: [PATCH 11/36] Update license tests to use default packages-exclude list (#24) * Update license tests to validate changes * Update license tests to use default branch --- .github/workflows/license_tests.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.github/workflows/license_tests.yml b/.github/workflows/license_tests.yml index 9469733c..7d0c4f6b 100644 --- a/.github/workflows/license_tests.yml +++ b/.github/workflows/license_tests.yml @@ -8,5 +8,3 @@ on: jobs: license_tests: uses: neongeckocom/.github/.github/workflows/license_tests.yml@master - with: - packages-exclude: '^(neon-data-models|dnspython|typing_extensions).*' From e971e779129ac43ee6efd4474e8d756c1ee4230f Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Tue, 27 May 2025 22:46:52 +0000 Subject: [PATCH 12/36] Increment Version to 0.0.2a6 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 5ca8c54b..bd827ead 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a5" +__version__ = "0.0.2a6" From 9d427e4362b202c47b955ce8207b1b6f1a186ff7 Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Fri, 30 May 2025 18:58:32 -0700 Subject: [PATCH 13/36] Add a validator to `LLMGetInferenceHttpRequest` to build `model` from `llm_name` and `llm_revision` if not otherwise specified (#25) --- neon_data_models/models/api/http/brainforge.py | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/neon_data_models/models/api/http/brainforge.py b/neon_data_models/models/api/http/brainforge.py index 7f099e3b..f397d112 100644 --- a/neon_data_models/models/api/http/brainforge.py +++ b/neon_data_models/models/api/http/brainforge.py @@ -25,7 +25,7 @@ # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. from typing import List, Optional -from pydantic import Field +from pydantic import Field, model_validator from neon_data_models.models.base import BaseModel from neon_data_models.models.api.llm import BrainForgeLLM, LLMPersona, LLMRequest @@ -48,8 +48,15 @@ class LLMGetPersonasHttpResponse(BaseModel): class LLMGetInferenceHttpRequest(LLMRequest): llm_name: str = Field(description="Model name to request") llm_revision: str = Field(description="Model revision to request") - model: Optional[str] = None + model: Optional[str] = Field( + default=None, + description="Model ID (@) used for vLLM inference") + @model_validator(mode='after') + def set_model_from_name_and_revision(self): + if self.model is None: + self.model = f"{self.llm_name}@{self.llm_revision}" + return self __all__ = [LLMGetModelsHttpResponse.__name__, LLMGetPersonasHttpRequest.__name__, From ec8fc5348ceb7f7dd732685296723406e90fa76a Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Sat, 31 May 2025 01:58:48 +0000 Subject: [PATCH 14/36] Increment Version to 0.0.2a7 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index bd827ead..90182043 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a6" +__version__ = "0.0.2a7" From 390bb319239268388f5e89e8362a04ae1c72f224 Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Fri, 13 Jun 2025 15:40:28 -0700 Subject: [PATCH 15/36] Update `chatbots` MQ models to improve validation and backwards-compat (#21) * Update `chatbots` MQ models to resolve missing `proctor` data * Add test of raised excpetion per review --- neon_data_models/models/api/mq/chatbots.py | 8 +++- tests/models/api/test_mq.py | 43 ++++++++++++++++++++++ 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/neon_data_models/models/api/mq/chatbots.py b/neon_data_models/models/api/mq/chatbots.py index 9cf04d8a..333b3350 100644 --- a/neon_data_models/models/api/mq/chatbots.py +++ b/neon_data_models/models/api/mq/chatbots.py @@ -58,7 +58,7 @@ class ChatbotsMqRequest(KlatContext, MQContext): default= datetime.now(tz=timezone.utc), description="Timestamp when the shout was created") requested_participants: Optional[List[str]] = Field( - default=None, + default=None, alias="participating_subminds", description="List of CCAI participants requested to handle the shout") recipient: Optional[str] = Field( default=None, description="Explicitly defined recipient of the shout") @@ -150,6 +150,12 @@ class ChatbotsMqSubmindResponse(KlatContext, MQContext): default=CcaiState.IDLE, deprecated=True, alias="promptState", description="State of the CCAI conversation associated with the shout") + @model_validator(mode='after') + def set_username_from_user_id(self): + if self.username is None and self.user_id: + self.username = self.user_id.rsplit('-', 1)[0] + return self + @model_validator(mode='before') @classmethod def validate_inputs(cls, values): diff --git a/tests/models/api/test_mq.py b/tests/models/api/test_mq.py index d1084387..2981a04b 100644 --- a/tests/models/api/test_mq.py +++ b/tests/models/api/test_mq.py @@ -1610,6 +1610,49 @@ def test_chatbots_mq_submind_response(self): self.assertEqual(serialized["nick"], "submind1") # Check backwards compatibility field self.assertEqual(serialized["shout"], "Full test response") # Check backwards compatibility field + + # Test username from user_id with a single hyphen + hyphen_user_id = ChatbotsMqSubmindResponse( + userID="service-123456", + messageText="Test message", + message_id="test_id" + ) + self.assertEqual(hyphen_user_id.username, "service") + + # Test username from user_id with multiple hyphens + multiple_hyphens = ChatbotsMqSubmindResponse( + userID="service-name-with-hyphens-123456", + messageText="Test message", + message_id="test_id" + ) + self.assertEqual(multiple_hyphens.username, "service-name-with-hyphens") + + # Test username from user_id without hyphen + no_hyphen = ChatbotsMqSubmindResponse( + userID="servicename", + messageText="Test message", + message_id="test_id" + ) + self.assertEqual(no_hyphen.username, "servicename") + + # Test existing username is preserved when both fields are provided + preserved_username = ChatbotsMqSubmindResponse( + userID="service-123456", + username="custom_name", + messageText="Test message", + message_id="test_id" + ) + self.assertEqual(preserved_username.username, "custom_name") + + # Test when user_id is None + with self.assertRaises(ValidationError) as e: + none_user_id = ChatbotsMqSubmindResponse( + userID=None, + messageText="Test message", + message_id="test_id" + ) + self.assertEqual(str(e), "user_id cannot be None") + # Test missing required fields with self.assertRaises(ValidationError): ChatbotsMqSubmindResponse(messageText="Missing user_id", message_id="test_id") From 37d9bfdbb10b0798e242fda693117c4cb04beba0 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Fri, 13 Jun 2025 22:40:41 +0000 Subject: [PATCH 16/36] Increment Version to 0.0.2a8 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 90182043..4ae6b9fe 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a7" +__version__ = "0.0.2a8" From f57a5d999633676fc9f2689ff23afcd002c6728e Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Fri, 13 Jun 2025 15:40:54 -0700 Subject: [PATCH 17/36] Specify `discussion_rounds` in new prompt model (#23) --- neon_data_models/models/api/mq/chatbots.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/neon_data_models/models/api/mq/chatbots.py b/neon_data_models/models/api/mq/chatbots.py index 333b3350..8e4c55e7 100644 --- a/neon_data_models/models/api/mq/chatbots.py +++ b/neon_data_models/models/api/mq/chatbots.py @@ -255,6 +255,9 @@ class ChatbotsMqNewPrompt(ChatbotsMqSubmindResponse): prompt_state: CcaiState = Field( default=CcaiState.IDLE, deprecated=True, description="Implemented for backwards-compat. New Prompt always IDLE") + discussion_rounds: int = Field( + default=2, + description="Number of discussion rounds per cycle for this prompt") context: Optional[dict] = Field(default=None, deprecated=True, alias="conversation_context") From b8c139b86566da9536d41c06be491ac0f3ded774 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Fri, 13 Jun 2025 22:41:03 +0000 Subject: [PATCH 18/36] Increment Version to 0.0.2a9 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 4ae6b9fe..f847bc5a 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a8" +__version__ = "0.0.2a9" From 02a7c8adef0939a6035a8099c33383e9555c992c Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Thu, 7 Aug 2025 09:15:33 -0700 Subject: [PATCH 19/36] Implement OpenAI Completion request/response models with unit tests (#26) * Implement OpenAI Completion request/response models with unit tests * Include JSON schema for BrainForge endpoints to improve HANA documentation * Implement `extra_body` support with test coverage --- .../models/api/http/brainforge.py | 183 +++++++++++++++- neon_data_models/models/api/llm.py | 5 +- tests/models/api/test_http.py | 195 ++++++++++++++++++ 3 files changed, 371 insertions(+), 12 deletions(-) create mode 100644 tests/models/api/test_http.py diff --git a/neon_data_models/models/api/http/brainforge.py b/neon_data_models/models/api/http/brainforge.py index f397d112..6b450f18 100644 --- a/neon_data_models/models/api/http/brainforge.py +++ b/neon_data_models/models/api/http/brainforge.py @@ -24,11 +24,18 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -from typing import List, Optional +from typing import List, Optional, Dict, Literal, Any +from uuid import uuid4 +from time import time from pydantic import Field, model_validator from neon_data_models.models.base import BaseModel -from neon_data_models.models.api.llm import BrainForgeLLM, LLMPersona, LLMRequest +from neon_data_models.models.api.llm import ( + BrainForgeLLM, + LLMPersona, + LLMRequest, + LLMResponse, +) class LLMGetModelsHttpResponse(BaseModel): @@ -37,28 +44,182 @@ class LLMGetModelsHttpResponse(BaseModel): class LLMGetPersonasHttpRequest(BaseModel): model_id: str = Field( - description="Model ID (@) to get personas for") + description="Model ID (@) to get personas for" + ) class LLMGetPersonasHttpResponse(BaseModel): personas: List[LLMPersona] = Field( - description="List of personas associated with the requested model.") + description="List of personas associated with the requested model." + ) class LLMGetInferenceHttpRequest(LLMRequest): llm_name: str = Field(description="Model name to request") llm_revision: str = Field(description="Model revision to request") model: Optional[str] = Field( - default=None, - description="Model ID (@) used for vLLM inference") + default=None, + description="Model ID (@) used for vLLM inference", + ) - @model_validator(mode='after') + @model_validator(mode="after") def set_model_from_name_and_revision(self): if self.model is None: self.model = f"{self.llm_name}@{self.llm_revision}" return self -__all__ = [LLMGetModelsHttpResponse.__name__, - LLMGetPersonasHttpRequest.__name__, - LLMGetPersonasHttpResponse.__name__, - LLMGetInferenceHttpRequest.__name__] + +class OpenAiCompletionRequest(BaseModel): + model: str = Field(description="Model ID (generally @)") + messages: List[Dict[Literal["role", "content"], str]] = Field( + description="List of messages to send to the model for completion, " + "including `system` and other context messages." + ) + max_tokens: Optional[int] = Field( + default=512, description="Maximum number of tokens to generate" + ) + temperature: Optional[float] = Field( + default=0.0, description="Temperature of generation" + ) + stream: Optional[bool] = Field( + default=False, description="Not Implemented." + ) + persona: Optional[LLMPersona] = Field(default=None) + extra_body: Optional[Dict[str, Any]] = Field(default=None) + + model_config = { + "json_schema_extra": { + "examples": [ + { + "model": "@", + "messages": [ + { + "role": "system", + "content": "You are a helpful assistant.", + }, + {"role": "user", "content": "Who are you?"}, + ], + "max_tokens": 512, + "temperature": 0.0, + "stream": False, + "extra_body": { + "use_beam_search": True, + "best_of": 3, + } + } + ] + } + } + + @model_validator(mode="after") + def ensure_default_values(self): + """Set default values for compat with BrainForge API.""" + self.max_tokens = self.max_tokens or 512 + self.temperature = self.temperature or 0.0 + if self.stream is None: + self.stream = False + if self.extra_body is None: + self.extra_body = {} + return self + + @model_validator(mode="after") + def get_persona(self): + """Determine a persona based on message history or default vanilla.""" + for message in self.messages: + if message["role"] == "system": + sys_message = self.messages.pop(self.messages.index(message)) + if sys_message["content"]: + self.persona = LLMPersona( + name="custom", system_prompt=sys_message["content"] + ) + return self + break + self.persona = LLMPersona(name="vanilla", system_prompt=None) + return self + + @model_validator(mode="after") + def validate_messages(self): + if len(self.messages) < 1: + raise ValueError("At least one `user` message is required.") + return self + + def to_llm_inference_http_request(self) -> LLMGetInferenceHttpRequest: + """ + Convert this OpenAI completion request to LLMGetInferenceHttpRequest + """ + model, revision = self.model.split("@", 1) + query = self.messages.pop(-1)["content"] + history = [(msg["role"], msg["content"]) for msg in self.messages] + return LLMGetInferenceHttpRequest( + llm_name=model, + llm_revision=revision, + model=self.model, + max_tokens=self.max_tokens, + temperature=self.temperature, + query=query, + history=history, + persona=self.persona, + **self.extra_body + ) + + +class OpenAiCompletionResponse(BaseModel): + id: str = Field( + description="UID for this completion request", + default_factory=lambda: uuid4().hex, + ) + object: Literal["chat.completion"] = "chat.completion" + created: float = Field( + description="Timestamp of completion request", + default_factory=lambda: time(), + ) + model: str = Field(description="Model ID used for this completion") + choices: List[ + Dict[Literal["message"], Dict[Literal["role", "content"], str]] + ] = Field(description="List of responses from the model") + + model_config = { + "json_schema_extra": { + "examples": [ + { + "id": uuid4().hex, + "object": "chat.completion", + "created": time(), + "model": "@", + "choices": [ + { + "message": { + "role": "assistant", + "content": "This is a sample response", + } + } + ], + } + ] + } + } + + @classmethod + def from_llm_response( + cls, llm_response: LLMResponse, llm_request: OpenAiCompletionRequest + ) -> "OpenAiCompletionResponse": + """Get an OpenAI response from a BrainForge LLMResponse.""" + choices = [ + { + "message": { + "role": "assistant", + "content": llm_response.response, + } + } + ] + return cls(model=llm_request.model, choices=choices) + + +__all__ = [ + LLMGetModelsHttpResponse.__name__, + LLMGetPersonasHttpRequest.__name__, + LLMGetPersonasHttpResponse.__name__, + LLMGetInferenceHttpRequest.__name__, + OpenAiCompletionRequest.__name__, + OpenAiCompletionResponse.__name__, +] diff --git a/neon_data_models/models/api/llm.py b/neon_data_models/models/api/llm.py index 10820125..e8e460e0 100644 --- a/neon_data_models/models/api/llm.py +++ b/neon_data_models/models/api/llm.py @@ -121,6 +121,9 @@ def validate_inputs(cls, values): for idx, itm in enumerate(values.get('history', [])): if itm[0] == "assistant": values['history'][idx] = ("llm", itm[1]) + # OpenAI `extra_body` may be included in input; parse those inputs + if values.get('use_beam_search') is not None: + values['beam_search'] = values['use_beam_search'] return values @model_validator(mode='after') @@ -161,7 +164,7 @@ def validate_request(self): # If beam search is enabled, temperature must be set to 0.0 if self.beam_search: - assert self.temperature == 0.0 + assert self.temperature == 0.0, "Beam search requires temperature 0" return self @property diff --git a/tests/models/api/test_http.py b/tests/models/api/test_http.py new file mode 100644 index 00000000..b9a4fb9e --- /dev/null +++ b/tests/models/api/test_http.py @@ -0,0 +1,195 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2025 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from unittest import TestCase +from pydantic import ValidationError + +from neon_data_models.models.api.llm import LLMPersona + + +class TestBrainForgeHttp(TestCase): + def test_llm_get_inference_http_request(self): + from neon_data_models.models.api.http.brainforge import ( + LLMGetInferenceHttpRequest, + ) + + # Valid request + valid_request = LLMGetInferenceHttpRequest( + llm_name="test_model", + llm_revision="1.0", + query="What is the weather like today?", + persona=LLMPersona(name="vanilla"), + history=[], + max_tokens=64, + temperature=0.7, + top_p=0.9, + stop_sequences=["\n"], + personas=["default"], + ) + self.assertEqual(valid_request.llm_name, "test_model") + self.assertEqual(valid_request.llm_revision, "1.0") + self.assertEqual(valid_request.model, "test_model@1.0") + self.assertEqual( + valid_request.query, "What is the weather like today?" + ) + + # Invalid request (missing required llm_name) + with self.assertRaises(ValidationError): + LLMGetInferenceHttpRequest( + llm_revision="1.0", query="What is the weather like today?" + ) + + def test_openai_completion_request(self): + from neon_data_models.models.api.http.brainforge import ( + OpenAiCompletionRequest, + ) + + # Valid no persona + valid_request_vanilla_persona = OpenAiCompletionRequest( + model="test_model@v1.0", + messages=[ + {"role": "system", "content": ""}, + {"role": "user", "content": "Hello, how are you?"}, + ], + max_tokens=None, + temperature=None, + stream=None, + ) + self.assertEqual( + valid_request_vanilla_persona.model, "test_model@v1.0" + ) + self.assertIsInstance(valid_request_vanilla_persona.max_tokens, int) + self.assertIsInstance(valid_request_vanilla_persona.temperature, float) + self.assertIsInstance(valid_request_vanilla_persona.stream, bool) + self.assertIsInstance( + valid_request_vanilla_persona.persona, LLMPersona + ) + self.assertEqual(valid_request_vanilla_persona.persona.name, "vanilla") + self.assertEqual(len(valid_request_vanilla_persona.messages), 1) + + as_llm_request = ( + valid_request_vanilla_persona.to_llm_inference_http_request() + ) + self.assertEqual(as_llm_request.history, []) + self.assertEqual(as_llm_request.query, "Hello, how are you?") + self.assertEqual( + as_llm_request.model, valid_request_vanilla_persona.model + ) + + # Valid with system prompt + valid_request_custom_persona = OpenAiCompletionRequest( + model="test_model@v1.0", + messages=[ + {"role": "system", "content": "Custom system prompt"}, + {"role": "user", "content": "Hello, how are you?"}, + {"role": "assistant", "content": "I'm fine, thank you!"}, + {"role": "user", "content": "who are you?"}, + ], + max_tokens=2048, + temperature=1.0, + stream=False, + ) + self.assertIsInstance(valid_request_custom_persona.persona, LLMPersona) + self.assertEqual(valid_request_custom_persona.persona.name, "custom") + self.assertEqual( + valid_request_custom_persona.persona.system_prompt, + "Custom system prompt", + ) + self.assertEqual(len(valid_request_custom_persona.messages), 3) + + as_llm_request = ( + valid_request_custom_persona.to_llm_inference_http_request() + ) + self.assertEqual(len(as_llm_request.history), 2) + self.assertEqual(as_llm_request.query, "who are you?") + self.assertEqual( + as_llm_request.model, valid_request_custom_persona.model + ) + + # Valid without system prompt + valid_request_no_system = OpenAiCompletionRequest( + model="test_model@v1.0", + messages=[ + {"role": "user", "content": "Hello, how are you?"}, + {"role": "assistant", "content": "I'm fine, thank you!"}, + {"role": "user", "content": "who are you?"}, + ], + max_tokens=1024, + temperature=0.0, + stream=False, + extra_body={"use_beam_search": True, + "best_of": 3}, + ) + self.assertIsInstance(valid_request_no_system.persona, LLMPersona) + self.assertEqual(valid_request_no_system.persona.name, "vanilla") + self.assertEqual(len(valid_request_no_system.messages), 3) + + as_llm_request = ( + valid_request_no_system.to_llm_inference_http_request() + ) + self.assertEqual(len(as_llm_request.history), 2) + self.assertEqual(as_llm_request.query, "who are you?") + self.assertEqual(as_llm_request.model, valid_request_no_system.model) + self.assertTrue(as_llm_request.beam_search) + + # Invalid no history + with self.assertRaises(ValidationError): + OpenAiCompletionRequest( + model="test_model@v1.0", + messages=[], + max_tokens=1024, + temperature=0.5, + stream=False, + ) + + def test_openai_completion_response(self): + from neon_data_models.models.api.http.brainforge import ( + OpenAiCompletionRequest, + OpenAiCompletionResponse, + ) + from neon_data_models.models.api.llm import LLMResponse + + # Valid response + llm_response = LLMResponse( + response="Hello", history=[("user", "hi"), ("llm", "Hello")] + ) + oai_request = OpenAiCompletionRequest( + model="model_name@revision", + messages=[{"role": "user", "content": "hi"}], + ) + + response = OpenAiCompletionResponse.from_llm_response( + llm_response=llm_response, llm_request=oai_request + ) + self.assertIsInstance(response.id, str) + self.assertEqual(response.object, "chat.completion") + self.assertIsInstance(response.created, float) + self.assertEqual(response.model, oai_request.model) + self.assertEqual(response.choices[0]["message"]["role"], "assistant") + self.assertEqual( + response.choices[0]["message"]["content"], llm_response.response + ) + From 86ce303f33aa8440f6b66c96943d4df93fd6e0b2 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Thu, 7 Aug 2025 16:15:47 +0000 Subject: [PATCH 20/36] Increment Version to 0.0.2a10 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index f847bc5a..e27a31d3 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a9" +__version__ = "0.0.2a10" From 9827dfad8d9e4b77efb993f2036085671366832f Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Fri, 15 Aug 2025 08:54:43 -0700 Subject: [PATCH 21/36] Implement models for the Skill API (#27) * Implement classes for Skills API messages in messagebus and mq modules * Troubleshooting MQ discriminator issue * Implement `neon.skill_api.query` MQ message with `as_messagebus_message` method to get Messagebus API-compatible `Message` * Refactor "neon.skill_api.query" to "neon.skill_api.call" for clarity * Refactor Skill API model structure and implement HTTP models * Fix missed refactor * Implement Http Response using `RootModel` to resolve exception * Refactor `NeonHttpListResponse` to `NeonHttpListSkillApiResponse` for clarity * Refactor param name for consistency * Address review comments * Add JSON schema to new HTTP request/response models * Fix JSON example in NeonHttpListSkillApiResponse model * Refactor Skill API HTTP response to account for refactor in HANA * Update NeonHttpListSkillApiResponse JSON example to a modern return value * Fix example schema * Add unit test coverage implemented with Claude * Fix copyright notice year Co-authored-by: Mike --------- Co-authored-by: Mike --- neon_data_models/models/api/http/neon.py | 137 ++++++ .../models/api/messagebus/__init__.py | 197 ++++++--- neon_data_models/models/api/mq/neon.py | 216 +++++++--- tests/models/api/test_http.py | 235 +++++++++++ tests/models/api/test_messagebus.py | 391 ++++++++++++++++++ tests/models/api/test_mq.py | 74 ++++ 6 files changed, 1143 insertions(+), 107 deletions(-) create mode 100644 neon_data_models/models/api/http/neon.py diff --git a/neon_data_models/models/api/http/neon.py b/neon_data_models/models/api/http/neon.py new file mode 100644 index 00000000..cdf22c83 --- /dev/null +++ b/neon_data_models/models/api/http/neon.py @@ -0,0 +1,137 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2025 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from typing import List +from pydantic import Field, RootModel +from neon_data_models.models.api.messagebus import ( + NeonSkillApiData, + SkillApiRequestData, + SkillApiResponseData, +) + + +class NeonSkillApiHttpData(NeonSkillApiData): + skill_id: str = Field( + description="ID of the skill providing this API method" + ) + api_method: str = Field(description="API method name") + + +class NeonHttpListSkillApiResponse(RootModel): + root: List[NeonSkillApiHttpData] + + model_config = { + "json_schema_extra": { + "examples": [ + [ + { + "help": "\n API Method to build a list of examples as listed in skill metadata.\n ", + "request_schema": None, + "response_schema": None, + "signature": None, + "type": "skill-about.neongeckocom.skill_info_examples", + "skill_id": "skill-about.neongeckocom", + "api_method": "skill_info_examples", + }, + { + "help": "\n Get the current timestamp in seconds since epoch.\n :param request: Request containing location to get time of\n :returns: Response containing current timestamp\n ", + "request_schema": { + "properties": { + "location": { + "anyOf": [ + {"type": "string"}, + {"type": "null"}, + ], + "default": None, + "title": "Location", + } + }, + "title": "_CurrentTimeRequest", + "type": "object", + }, + "response_schema": { + "properties": { + "current_timestamp": { + "title": "Current Timestamp", + "type": "number", + } + }, + "required": ["current_timestamp"], + "title": "_CurrentTimeResponse", + "type": "object", + }, + "signature": "(request: skill_date_time._CurrentTimeRequest) -> skill_date_time._CurrentTimeResponse", + "type": "skill-date_time.neongeckocom.get_current_time", + "skill_id": "skill-date_time.neongeckocom", + "api_method": "get_current_time", + }, + ] + ] + } + } + + +class NeonHttpSkillApiRequest(SkillApiRequestData): + skill_id: str = Field(description="skill_id being requested") + api_method: str = Field(description="API method being requested") + + model_config = { + "json_schema_extra": { + "examples": [ + { + "skill_id": "skill-about.neongeckocom", + "api_method": "skill_info_examples", + "args": [], + "kwargs": {}, + } + ] + } + } + + +class NeonHttpSkillApiResponse(SkillApiResponseData): + """ + Convenience wrapper to implement examples for Swagger UI + """ + + model_config = { + "json_schema_extra": { + "examples": [ + { + "result": "API Response matching advertised schema", + "error": None, + }, + {"result": None, "error": "API Method error message"}, + ] + } + } + + +__all__ = [ + NeonHttpListSkillApiResponse.__name__, + NeonHttpSkillApiRequest.__name__, + NeonHttpSkillApiResponse.__name__, +] diff --git a/neon_data_models/models/api/messagebus/__init__.py b/neon_data_models/models/api/messagebus/__init__.py index 2b4a6cb0..2f25326a 100644 --- a/neon_data_models/models/api/messagebus/__init__.py +++ b/neon_data_models/models/api/messagebus/__init__.py @@ -38,96 +38,109 @@ # Data models class GetTtsData(BaseModel): text: str = Field(description="Text to be spoken") - lang: str = Field(default="en-us", - description="BCP-47 language code for TTS") + lang: str = Field( + default="en-us", description="BCP-47 language code for TTS" + ) - @model_validator(mode='before') + @model_validator(mode="before") @classmethod def validate_inputs(cls, values): # Ensure input values are normalized to valid field names - if hasattr(values, 'model_dump'): + if hasattr(values, "model_dump"): values = values.model_dump() - if 'text' not in values: - values['text'] = values.get('utterance') + if "text" not in values: + values["text"] = values.get("utterance") return values class TtsResponse(BaseModel): sentence: str = Field(description="Text to be spoken") translated: bool = Field( - default=False, - description="True if the text has been translated before TTS synthesis") + default=False, + description="True if the text has been translated before TTS synthesis", + ) phonemes: Optional[str] = Field( - default=None, description="Phoneme representation of the sentence") + default=None, description="Phoneme representation of the sentence" + ) genders: List[Gender] = Field( description="List of genders included in the response `audio` field", - deprecated=True) - male: Optional[str] = Field(default=None, - description="Path to audio file in male voice") + deprecated=True, + ) + male: Optional[str] = Field( + default=None, description="Path to audio file in male voice" + ) female: Optional[str] = Field( - default=None, description="Path to audio file in female voice") + default=None, description="Path to audio file in female voice" + ) audio: Dict[Gender, str] - - @model_validator(mode='after') + + @model_validator(mode="after") def validate_gender_audio_match(self): """Validate that the genders list matches the audio dictionary keys.""" if not set(self.genders) == set(self.audio.keys()): raise ValueError( "All genders listed in 'genders' must have corresponding " - "entries in 'audio'") + "entries in 'audio'" + ) return self class TtsSpeaker(BaseModel): name: str = Field(default="Neon", description="Name of the speaker") speaker: Optional[str] = Field( - default=None, + default=None, description="Deprecated: Use 'name' instead", - deprecated=True) + deprecated=True, + ) language: str = Field(default="en-us", description="BCP-47 language code") - gender: Gender = Field(default="female", - description="Requested or synthesized gender") + gender: Gender = Field( + default="female", description="Requested or synthesized gender" + ) voice: Optional[str] = Field( - default=None, description="Requested or synthesized voice name") - - @model_validator(mode='before') + default=None, description="Requested or synthesized voice name" + ) + + @model_validator(mode="before") @classmethod def map_speaker_to_name(cls, values): - if hasattr(values, 'model_dump'): + if hasattr(values, "model_dump"): values = values.model_dump() - if 'name' not in values and 'speaker' in values: - values['name'] = values['speaker'] + if "name" not in values and "speaker" in values: + values["name"] = values["speaker"] return values class TtsReponseData(BaseModel): responses: Dict[str, TtsResponse] = Field( - description="Dictionary of BCP-47 lang codes to TTS responses") + description="Dictionary of BCP-47 lang codes to TTS responses" + ) speaker: Optional[TtsSpeaker] = Field( - default=None, description="Optional speaker metadata") + default=None, description="Optional speaker metadata" + ) class GetSttData(BaseModel): audio_data: str = Field(description="Base64-encoded audio data") - lang: str = Field(default="en-us", - description="BCP-47 language code for STT") + lang: str = Field( + default="en-us", description="BCP-47 language code for STT" + ) - @model_validator(mode='before') + @model_validator(mode="before") @classmethod def validate_inputs(cls, values): # Ensure input values are normalized to valid field names - if hasattr(values, 'model_dump'): + if hasattr(values, "model_dump"): values = values.model_dump() - if 'audio_data' not in values: - values['audio_data'] = values.get('message_body') + if "audio_data" not in values: + values["audio_data"] = values.get("message_body") return values class SttReponseData(BaseModel): transcripts: List[str] parser_data: Dict[str, Any] - - @model_validator(mode='after') + + @model_validator(mode="after") def validate_transcripts_not_empty(self): """Validate that the transcripts list is not empty.""" if not self.transcripts: @@ -137,17 +150,18 @@ def validate_transcripts_not_empty(self): class GetResponseData(BaseModel): utterances: List[str] = Field(description="List of input utterance(s)") - lang: str = Field(default="en-us", - description="BCP-47 language code for input/response") + lang: str = Field( + default="en-us", description="BCP-47 language code for input/response" + ) - @model_validator(mode='before') + @model_validator(mode="before") @classmethod def validate_inputs(cls, values): # Ensure input values are normalized to valid field names - if hasattr(values, 'model_dump'): + if hasattr(values, "model_dump"): values = values.model_dump() - if 'utterances' not in values: - values['utterances'] = [values.pop('messageText', '')] + if "utterances" not in values: + values["utterances"] = [values.pop("messageText", "")] return values @@ -162,7 +176,7 @@ class NeonGetTts(BaseMessage): msg_type: Literal["neon.get_tts"] = "neon.get_tts" data: GetTtsData - @model_validator(mode='after') + @model_validator(mode="after") def ensure_audio_destination(self): if "audio" not in self.context.destination: self.context.destination = ["audio"] @@ -173,7 +187,7 @@ class NeonGetStt(BaseMessage): msg_type: Literal["neon.get_stt"] = "neon.get_stt" data: GetSttData - @model_validator(mode='after') + @model_validator(mode="after") def ensure_audio_destination(self): if "audio" not in self.context.destination: # Default context is not valid for this request, so override it @@ -182,10 +196,12 @@ def ensure_audio_destination(self): class NeonTextInput(BaseMessage): - msg_type: Literal["recognizer_loop:utterance"] = "recognizer_loop:utterance" + msg_type: Literal["recognizer_loop:utterance"] = ( + "recognizer_loop:utterance" + ) data: GetResponseData - @model_validator(mode='after') + @model_validator(mode="after") def ensure_skills_destination(self): if "skills" not in self.context.destination: self.context.destination.append("skills") @@ -196,7 +212,7 @@ class NeonAudioInput(BaseMessage): msg_type: Literal["neon.audio_input"] = "neon.audio_input" data: GetSttData - @model_validator(mode='after') + @model_validator(mode="after") def ensure_audio_destination(self): if "audio" not in self.context.destination: # Default context is not valid for this request, so override it @@ -210,21 +226,92 @@ class NeonSttResponse(BaseMessage): class NeonTtsResponse(BaseMessage): - msg_type: Literal["neon.get_tts.response", - "klat.response"] = "neon.get_tts.response" + msg_type: Literal["neon.get_tts.response", "klat.response"] = ( + "neon.get_tts.response" + ) data: TtsReponseData class NeonGetLanguages(BaseMessage): msg_type: Literal["neon.languages.get"] = "neon.languages.get" - + class NeonLanguagesResponse(BaseMessage): - msg_type: Literal["neon.languages.get.response"] = "neon.languages.get.response" + msg_type: Literal["neon.languages.get.response"] = ( + "neon.languages.get.response" + ) data: NeonLanguagesData -__all__ = [NeonGetTts.__name__, NeonGetStt.__name__, NeonTextInput.__name__, - NeonAudioInput.__name__, NeonSttResponse.__name__, - NeonTtsResponse.__name__, NeonGetLanguages.__name__, - NeonLanguagesResponse.__name__] +class NeonGetSkillsApi(BaseMessage): + msg_type: Literal["neon.skill_api.get"] = "neon.skill_api.get" + + +class NeonSkillApiData(BaseModel): + help: str = Field(description="API Method docstring") + request_schema: Optional[dict] = Field( + default=None, description="JSON request schema for the API method" + ) + response_schema: Optional[dict] = Field( + default=None, description="JSON response schema for the API method" + ) + signature: Optional[str] = Field( + default=None, description="Python signature of the API method" + ) + msg_type: str = Field( + alias="type", + description="Message type associated with this API method", + ) + + +class NeonSkillsApiResponse(BaseMessage): + msg_type: Literal["neon.skill_api.get.response"] = ( + "neon.skill_api.get.response" + ) + data: Dict[str, Dict[str, NeonSkillApiData]] = Field( + description="Dict of skill_id to API methods" + ) + + +class SkillApiRequestData(BaseModel): + args: List[Any] = Field(default=[], description="Positional arguments") + kwargs: Dict[str, Any] = Field(default={}, description="Keyword arguments") + + +class SkillApiResponseData(BaseModel): + result: Any = Field( + description="Result of the API method call", + default=None, + ) + error: Optional[str] = Field( + default=None, description="Error message if the call failed" + ) + + +class NeonCallSkillApi(BaseMessage): + msg_type: str = Field(description="Requested API Message type") + data: SkillApiRequestData = Field( + default=SkillApiRequestData(), + description="Data for the API method call", + ) + + +class NeonCallSkillApiResponse(BaseMessage): + msg_type: str = Field(description="Requested API response Message type") + data: SkillApiResponseData = Field(description="API Method response data") + + +__all__ = [ + NeonGetTts.__name__, + NeonGetStt.__name__, + NeonTextInput.__name__, + NeonAudioInput.__name__, + NeonSttResponse.__name__, + NeonTtsResponse.__name__, + NeonGetLanguages.__name__, + NeonLanguagesResponse.__name__, + NeonGetSkillsApi.__name__, + NeonSkillsApiResponse.__name__, + NeonCallSkillApi.__name__, + NeonCallSkillApiResponse.__name__, +] diff --git a/neon_data_models/models/api/mq/neon.py b/neon_data_models/models/api/mq/neon.py index e22242ac..ddd9cc53 100644 --- a/neon_data_models/models/api/mq/neon.py +++ b/neon_data_models/models/api/mq/neon.py @@ -25,55 +25,77 @@ # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. from typing import Annotated, List, Literal, Union +from ovos_bus_client.message import Message from pydantic import Field, TypeAdapter, model_validator from neon_data_models.models.base import BaseModel -from neon_data_models.models.base.contexts import KlatContext, MQContext, \ - SessionContext, TimingContext +from neon_data_models.models.base.contexts import ( + KlatContext, + MQContext, + SessionContext, + TimingContext, +) from neon_data_models.models.base.messagebus import BaseMessage, MessageContext from neon_data_models.models.user.neon_profile import UserProfile -from neon_data_models.models.api.messagebus import NeonGetLanguages, NeonGetTts, NeonGetStt, \ - NeonAudioInput, NeonLanguagesResponse, NeonTextInput, NeonSttResponse, NeonTtsResponse +from neon_data_models.models.api.messagebus import ( + NeonGetLanguages, + NeonGetTts, + NeonGetStt, + NeonAudioInput, + NeonLanguagesResponse, + NeonTextInput, + NeonSttResponse, + NeonTtsResponse, + NeonGetSkillsApi, + NeonSkillsApiResponse, + NeonCallSkillApi, + NeonCallSkillApiResponse, + SkillApiRequestData +) class GetTtsData(BaseModel): text: str = Field(description="Text to be spoken") - lang: str = Field(default="en-us", - description="BCP-47 language code for TTS") + lang: str = Field( + default="en-us", description="BCP-47 language code for TTS" + ) - @model_validator(mode='before') + @model_validator(mode="before") @classmethod def validate_inputs(cls, values): - if 'text' not in values: - values['text'] = values.pop('utterance', None) + if "text" not in values: + values["text"] = values.pop("utterance", None) return values class GetSttData(BaseModel): audio_data: str = Field(description="Base64-encoded audio data") - lang: str = Field(default="en-us", - description="BCP-47 language code for STT") + lang: str = Field( + default="en-us", description="BCP-47 language code for STT" + ) - @model_validator(mode='before') + @model_validator(mode="before") @classmethod def validate_inputs(cls, values): - if 'audio_data' not in values: - values['audio_data'] = values.get('message_body') + if "audio_data" not in values: + values["audio_data"] = values.get("message_body") return values class GetResponseData(BaseModel): utterances: List[str] = Field(description="List of input utterance(s)") - lang: str = Field(default="en-us", - description="BCP-47 language code for input/response") + lang: str = Field( + default="en-us", description="BCP-47 language code for input/response" + ) - @model_validator(mode='before') + @model_validator(mode="before") @classmethod def validate_inputs(cls, values): - if 'utterances' not in values: - values['utterances'] = [values.pop('messageText', '')] + if "utterances" not in values: + values["utterances"] = [values.pop("messageText", "")] return values + class NeonMqGetTts(NeonGetTts, MQContext): """ Data model for an MQ message requesting TTS synthesis @@ -122,6 +144,59 @@ class NeonMqLanguagesResponse(NeonLanguagesResponse, MQContext): """ +class NeonMqGetSkillsApi(NeonGetSkillsApi, MQContext): + """ + Data model for an MQ message requesting available skill APIs + """ + + +class NeonMqSkillsApiResponse(NeonSkillsApiResponse, MQContext): + """ + Data model for an MQ message containing available skill APIs + """ + + +class NeonMqCallSkillApi(NeonCallSkillApi, MQContext): + """ + Data model for an MQ message calling a skill API + """ + + class _RequestData(SkillApiRequestData): + msg_type: str = Field( + description="Message type associated with the specific API requested" + ) + + msg_type: Literal["neon.skill_api.call"] = Field( + default="neon.skill_api.call", + description="Message type for skill API calls", + ) + data: _RequestData = Field( + description="API request data including `msg_type` and call params" + ) + + def as_messagebus_message(self) -> Message: + """ + Override default Message translation to account for `msg_type` being + specified in request data + """ + return Message( + msg_type=self.data.msg_type, + data={"args": self.data.args, "kwargs": self.data.kwargs}, + context=self.context.model_dump() + ) + +class NeonMqCallSkillApiResponse(NeonCallSkillApiResponse, MQContext): + """ + Data model for an MQ message containing a skill API response. + Note that `msg_type` is overridden from the internal API-specific value + to a generic type for MQ routing. + """ + msg_type: Literal["neon.skill_api.response"] = Field( + default="neon.skill_api.response", + description="Message type for skill API responses", + ) + + class NeonMqUnknownMessage(BaseMessage, MQContext): """ Default message class for validating Messagebus messages that should be @@ -134,21 +209,35 @@ class NeonApiMessage: Type adapter for validating an arbitrary MQ message. This will always return an instance that extends `BaseMessage` and `MQContext`. """ - ta = TypeAdapter(Annotated[Union[NeonMqGetStt, NeonMqGetTts, - NeonMqTextInput, NeonMqSttResponse, - NeonMqTtsResponse, NeonMqGetLanguages, - NeonMqLanguagesResponse], - Field(discriminator='msg_type')]) + + ta = TypeAdapter( + Annotated[ + Union[ + NeonMqGetStt, + NeonMqGetTts, + NeonMqTextInput, + NeonMqSttResponse, + NeonMqTtsResponse, + NeonMqGetLanguages, + NeonMqLanguagesResponse, + NeonMqGetSkillsApi, + NeonMqSkillsApiResponse, + NeonMqCallSkillApi, + NeonMqCallSkillApiResponse, + ], + Field(discriminator="msg_type"), + ] + ) @classmethod def __new__(cls, *args, **kwargs) -> BaseMessage: # Parse the MQ Context from a `Message` input to create a proper API message - if 'message_id' not in kwargs: + if "message_id" not in kwargs: # Extract MQ context data from the message - mq_data = kwargs.get('context', {}).get('mq', {}) + mq_data = kwargs.get("context", {}).get("mq", {}) # Update values with MQ context data kwargs.update(mq_data) - + try: return cls.ta.validate_python(kwargs) except Exception as e: @@ -160,38 +249,61 @@ def from_sio_message(sio_message: dict) -> BaseMessage: """ Parse a Klat SocketIO message into a valid MQ API message """ - requested_service = sio_message.get("requested_skill", - "recognizer").lower() + requested_service = sio_message.get( + "requested_skill", "recognizer" + ).lower() if requested_service not in ["stt", "tts", "recognizer"]: - raise ValueError(f"Invalid requested service '{requested_service}'") + raise ValueError( + f"Invalid requested service '{requested_service}'" + ) klat_context = KlatContext(**sio_message) mq_context = MQContext(**sio_message) - context = MessageContext(source="mq_api", - client=sio_message.get("client", "unknown"), - username=sio_message.get("nick", "guest"), - klat_data=klat_context, mq=mq_context, - user_profiles=[UserProfile()], - session=SessionContext( - session_id=sio_message.get("cid", "klat")), - timing=TimingContext( - client_sent=sio_message.get("timeCreated")) + context = MessageContext( + source="mq_api", + client=sio_message.get("client", "unknown"), + username=sio_message.get("nick", "guest"), + klat_data=klat_context, + mq=mq_context, + user_profiles=[UserProfile()], + session=SessionContext(session_id=sio_message.get("cid", "klat")), + timing=TimingContext(client_sent=sio_message.get("timeCreated")), ) if requested_service == "stt": context.destination = ["speech"] - return NeonMqGetStt(data=GetSttData(**sio_message), context=context, - **mq_context.model_dump()) + return NeonMqGetStt( + data=GetSttData(**sio_message), + context=context, + **mq_context.model_dump(), + ) elif requested_service == "tts": context.destination = ["audio"] - return NeonMqGetTts(data=GetTtsData(**sio_message), context=context, - **mq_context.model_dump()) + return NeonMqGetTts( + data=GetTtsData(**sio_message), + context=context, + **mq_context.model_dump(), + ) elif requested_service == "recognizer": context.destination = ["skills"] - return NeonMqTextInput(data=GetResponseData(**sio_message), - context=context, **mq_context.model_dump()) - - -__all__ = [NeonMqGetTts.__name__, NeonMqGetStt.__name__, - NeonMqTextInput.__name__, NeonMqAudioInput.__name__, - NeonMqSttResponse.__name__, NeonMqTtsResponse.__name__, - NeonMqGetLanguages.__name__, NeonMqLanguagesResponse.__name__, - NeonApiMessage.__name__, NeonMqUnknownMessage.__name__] + return NeonMqTextInput( + data=GetResponseData(**sio_message), + context=context, + **mq_context.model_dump(), + ) + + +__all__ = [ + NeonMqGetTts.__name__, + NeonMqGetStt.__name__, + NeonMqTextInput.__name__, + NeonMqAudioInput.__name__, + NeonMqSttResponse.__name__, + NeonMqTtsResponse.__name__, + NeonMqGetLanguages.__name__, + NeonMqLanguagesResponse.__name__, + NeonApiMessage.__name__, + NeonMqGetSkillsApi.__name__, + NeonMqSkillsApiResponse.__name__, + NeonMqCallSkillApi.__name__, + NeonMqCallSkillApiResponse.__name__, + NeonMqUnknownMessage.__name__, +] diff --git a/tests/models/api/test_http.py b/tests/models/api/test_http.py index b9a4fb9e..fc4424cc 100644 --- a/tests/models/api/test_http.py +++ b/tests/models/api/test_http.py @@ -193,3 +193,238 @@ def test_openai_completion_response(self): response.choices[0]["message"]["content"], llm_response.response ) + +class TestNeonHttp(TestCase): + def test_neon_skill_api_http_data(self): + from neon_data_models.models.api.http.neon import NeonSkillApiHttpData + + # Valid data with all fields + valid_data = NeonSkillApiHttpData( + skill_id="skill-about.neongeckocom", + api_method="skill_info_examples", + help="API Method to build a list of examples", + request_schema=None, + response_schema=None, + signature=None, + msg_type="skill-about.neongeckocom.skill_info_examples", + ) + self.assertEqual(valid_data.skill_id, "skill-about.neongeckocom") + self.assertEqual(valid_data.api_method, "skill_info_examples") + self.assertEqual(valid_data.help, "API Method to build a list of examples") + self.assertEqual( + valid_data.msg_type, "skill-about.neongeckocom.skill_info_examples" + ) + + # Valid data with schema + valid_data_with_schema = NeonSkillApiHttpData( + skill_id="skill-date_time.neongeckocom", + api_method="get_current_time", + help="Get the current timestamp in seconds since epoch.", + request_schema={ + "properties": { + "location": { + "anyOf": [{"type": "string"}, {"type": "null"}], + "default": None, + "title": "Location", + } + }, + "title": "_CurrentTimeRequest", + "type": "object", + }, + response_schema={ + "properties": { + "current_timestamp": { + "title": "Current Timestamp", + "type": "number", + } + }, + "required": ["current_timestamp"], + "title": "_CurrentTimeResponse", + "type": "object", + }, + signature="(request: skill_date_time._CurrentTimeRequest) -> skill_date_time._CurrentTimeResponse", + msg_type="skill-date_time.neongeckocom.get_current_time", + ) + self.assertIsInstance(valid_data_with_schema.request_schema, dict) + self.assertIsInstance(valid_data_with_schema.response_schema, dict) + self.assertIsInstance(valid_data_with_schema.signature, str) + + # Invalid data (missing required fields) + with self.assertRaises(ValidationError): + NeonSkillApiHttpData( + skill_id="test_skill", + # Missing api_method + help="Test help", + ) + + def test_neon_http_list_skill_api_response(self): + from neon_data_models.models.api.http.neon import ( + NeonHttpListSkillApiResponse, + NeonSkillApiHttpData, + ) + + # Valid response with multiple items + valid_response = NeonHttpListSkillApiResponse( + root=[ + NeonSkillApiHttpData( + skill_id="skill-about.neongeckocom", + api_method="skill_info_examples", + help="API Method to build a list of examples", + request_schema=None, + response_schema=None, + signature=None, + msg_type="skill-about.neongeckocom.skill_info_examples", + ), + NeonSkillApiHttpData( + skill_id="skill-date_time.neongeckocom", + api_method="get_current_time", + help="Get the current timestamp in seconds since epoch.", + request_schema={ + "properties": { + "location": { + "anyOf": [{"type": "string"}, {"type": "null"}], + "default": None, + "title": "Location", + } + }, + "title": "_CurrentTimeRequest", + "type": "object", + }, + response_schema={ + "properties": { + "current_timestamp": { + "title": "Current Timestamp", + "type": "number", + } + }, + "required": ["current_timestamp"], + "title": "_CurrentTimeResponse", + "type": "object", + }, + signature="(request: skill_date_time._CurrentTimeRequest) -> skill_date_time._CurrentTimeResponse", + msg_type="skill-date_time.neongeckocom.get_current_time", + ), + ] + ) + self.assertEqual(len(valid_response.root), 2) + self.assertIsInstance(valid_response.root[0], NeonSkillApiHttpData) + self.assertEqual( + valid_response.root[0].skill_id, "skill-about.neongeckocom" + ) + self.assertEqual( + valid_response.root[1].skill_id, "skill-date_time.neongeckocom" + ) + + # Valid empty response + empty_response = NeonHttpListSkillApiResponse(root=[]) + self.assertEqual(len(empty_response.root), 0) + + # Test serialization/deserialization + data = valid_response.model_dump() + # For RootModel, the data is a list, so we create directly from the list + reconstructed = NeonHttpListSkillApiResponse(root=data) + self.assertEqual(len(reconstructed.root), 2) + self.assertEqual( + reconstructed.root[0].skill_id, "skill-about.neongeckocom" + ) + with self.assertRaises(ValidationError): + # Invalid, empty response + NeonHttpListSkillApiResponse(root=None) + + with self.assertRaises(ValidationError): + # Invalid type + NeonHttpListSkillApiResponse(root={}) + + def test_neon_http_skill_api_request(self): + from neon_data_models.models.api.http.neon import NeonHttpSkillApiRequest + + # Valid request with no args/kwargs + valid_request = NeonHttpSkillApiRequest( + skill_id="skill-about.neongeckocom", + api_method="skill_info_examples", + ) + self.assertEqual(valid_request.skill_id, "skill-about.neongeckocom") + self.assertEqual(valid_request.api_method, "skill_info_examples") + self.assertEqual(valid_request.args, []) + self.assertEqual(valid_request.kwargs, {}) + + # Valid request with args and kwargs + valid_request_with_data = NeonHttpSkillApiRequest( + skill_id="skill-date_time.neongeckocom", + api_method="get_current_time", + args=["arg1", "arg2"], + kwargs={"location": "Seattle", "timezone": "UTC"}, + ) + self.assertEqual(valid_request_with_data.args, ["arg1", "arg2"]) + self.assertEqual( + valid_request_with_data.kwargs, + {"location": "Seattle", "timezone": "UTC"}, + ) + + # Invalid request (missing required fields) + with self.assertRaises(ValidationError): + NeonHttpSkillApiRequest( + skill_id="test_skill", + # Missing api_method + ) + + with self.assertRaises(ValidationError): + NeonHttpSkillApiRequest( + # Missing skill_id + api_method="test_method", + ) + + def test_neon_http_skill_api_response(self): + from neon_data_models.models.api.http.neon import NeonHttpSkillApiResponse + + # Valid response with result + valid_response_with_result = NeonHttpSkillApiResponse( + result="API Response matching advertised schema", + error=None, + ) + self.assertEqual( + valid_response_with_result.result, + "API Response matching advertised schema", + ) + self.assertIsNone(valid_response_with_result.error) + + # Valid response with error + valid_response_with_error = NeonHttpSkillApiResponse( + result=None, error="API Method error message" + ) + self.assertIsNone(valid_response_with_error.result) + self.assertEqual( + valid_response_with_error.error, "API Method error message" + ) + + # Valid response with complex result + complex_result = { + "timestamp": 1640995200.0, + "timezone": "UTC", + "formatted_time": "2022-01-01 00:00:00", + } + valid_response_complex = NeonHttpSkillApiResponse( + result=complex_result, error=None + ) + self.assertEqual(valid_response_complex.result, complex_result) + self.assertIsNone(valid_response_complex.error) + + # Valid response with list result + list_result = ["item1", "item2", "item3"] + valid_response_list = NeonHttpSkillApiResponse( + result=list_result, error=None + ) + self.assertEqual(valid_response_list.result, list_result) + + # Valid response with both result and error (should be allowed) + both_fields_response = NeonHttpSkillApiResponse( + result="some result", error="some error" + ) + self.assertEqual(both_fields_response.result, "some result") + self.assertEqual(both_fields_response.error, "some error") + + # Test serialization/deserialization + response_data = valid_response_complex.model_dump() + reconstructed = NeonHttpSkillApiResponse(**response_data) + self.assertEqual(reconstructed.result, complex_result) + self.assertIsNone(reconstructed.error) \ No newline at end of file diff --git a/tests/models/api/test_messagebus.py b/tests/models/api/test_messagebus.py index 9e91527c..d52bbef3 100644 --- a/tests/models/api/test_messagebus.py +++ b/tests/models/api/test_messagebus.py @@ -717,3 +717,394 @@ def test_stt_response_data_edge_cases(self): } stt_response = SttReponseData(**valid_empty_parser) self.assertEqual(len(stt_response.parser_data), 0) + + def test_neon_get_skills_api(self): + from neon_data_models.models.api.messagebus import NeonGetSkillsApi + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid message + valid_message = NeonGetSkillsApi(data={}, context={}) + self.assertIsInstance(valid_message, NeonGetSkillsApi) + self.assertIsInstance(valid_message, BaseMessage) + self.assertEqual(valid_message.msg_type, "neon.skill_api.get") + self.assertEqual(valid_message.data, {}) + + # Test with custom context + context_data = {"source": "test_client", "destination": ["skills"]} + message_with_context = NeonGetSkillsApi(data={}, context=context_data) + self.assertEqual(message_with_context.context.source, "test_client") + self.assertEqual(message_with_context.context.destination, ["skills"]) + + # Test with invalid msg_type (should fail due to Literal constraint) + with self.assertRaises(ValidationError): + NeonGetSkillsApi(data={}, context={}, msg_type="invalid.type") + + def test_neon_skill_api_data(self): + from neon_data_models.models.api.messagebus import NeonSkillApiData + + # Test valid data with all fields + valid_data = { + "help": "Get the current timestamp in seconds since epoch.", + "request_schema": { + "properties": { + "location": { + "anyOf": [{"type": "string"}, {"type": "null"}], + "default": None, + "title": "Location", + } + }, + "title": "_CurrentTimeRequest", + "type": "object", + }, + "response_schema": { + "properties": { + "current_timestamp": { + "title": "Current Timestamp", + "type": "number", + } + }, + "required": ["current_timestamp"], + "title": "_CurrentTimeResponse", + "type": "object", + }, + "signature": "(request: skill_date_time._CurrentTimeRequest) -> skill_date_time._CurrentTimeResponse", + "type": "skill-date_time.neongeckocom.get_current_time", + } + skill_api_data = NeonSkillApiData(**valid_data) + self.assertIsInstance(skill_api_data, NeonSkillApiData) + self.assertEqual(skill_api_data.help, "Get the current timestamp in seconds since epoch.") + self.assertEqual(skill_api_data.msg_type, "skill-date_time.neongeckocom.get_current_time") + self.assertIsInstance(skill_api_data.request_schema, dict) + self.assertIsInstance(skill_api_data.response_schema, dict) + self.assertIsNotNone(skill_api_data.signature) + + # Test valid data with minimal fields (only required fields) + minimal_data = { + "help": "Simple API method", + "type": "test.skill.method" + } + minimal_skill_api = NeonSkillApiData(**minimal_data) + self.assertEqual(minimal_skill_api.help, "Simple API method") + self.assertEqual(minimal_skill_api.msg_type, "test.skill.method") + self.assertIsNone(minimal_skill_api.request_schema) + self.assertIsNone(minimal_skill_api.response_schema) + self.assertIsNone(minimal_skill_api.signature) + + # Test missing required fields + with self.assertRaises(ValidationError): + NeonSkillApiData(help="Missing type field") + + with self.assertRaises(ValidationError): + NeonSkillApiData(type="missing.help.field") + + def test_neon_skills_api_response(self): + from neon_data_models.models.api.messagebus import NeonSkillsApiResponse, NeonSkillApiData + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid response with multiple skills + skill1_data = NeonSkillApiData( + help="Get current time", + msg_type="skill-date_time.neongeckocom.get_current_time", + request_schema={"type": "object"}, + response_schema={"type": "object"} + ) + skill2_data = NeonSkillApiData( + help="Get weather info", + msg_type="skill-weather.neongeckocom.get_weather" + ) + + response_data = { + "skill-date_time.neongeckocom": { + "get_current_time": skill1_data + }, + "skill-weather.neongeckocom": { + "get_weather": skill2_data + } + } + + valid_response = NeonSkillsApiResponse(data=response_data, context={}) + self.assertIsInstance(valid_response, NeonSkillsApiResponse) + self.assertIsInstance(valid_response, BaseMessage) + self.assertEqual(valid_response.msg_type, "neon.skill_api.get.response") + self.assertEqual(len(valid_response.data), 2) + self.assertIn("skill-date_time.neongeckocom", valid_response.data) + self.assertIn("skill-weather.neongeckocom", valid_response.data) + self.assertEqual( + valid_response.data["skill-date_time.neongeckocom"]["get_current_time"].help, + "Get current time" + ) + + # Test empty response + empty_response = NeonSkillsApiResponse(data={}, context={}) + self.assertEqual(len(empty_response.data), 0) + + # Test response with complex nested structure + complex_skill_data = { + "skill-complex.test": { + "method1": NeonSkillApiData(help="Method 1", msg_type="test.method1"), + "method2": NeonSkillApiData(help="Method 2", msg_type="test.method2"), + "method3": NeonSkillApiData(help="Method 3", msg_type="test.method3") + } + } + complex_response = NeonSkillsApiResponse(data=complex_skill_data, context={}) + self.assertEqual(len(complex_response.data["skill-complex.test"]), 3) + self.assertIn("method2", complex_response.data["skill-complex.test"]) + + # Test missing required fields + with self.assertRaises(ValidationError): + NeonSkillsApiResponse(context={}) + + def test_skill_api_request_data(self): + from neon_data_models.models.api.messagebus import SkillApiRequestData + + # Test valid data with no args/kwargs + valid_request = SkillApiRequestData() + self.assertIsInstance(valid_request, SkillApiRequestData) + self.assertEqual(valid_request.args, []) + self.assertEqual(valid_request.kwargs, {}) + + # Test valid data with args and kwargs + valid_request_with_data = SkillApiRequestData( + args=["arg1", "arg2", 123], + kwargs={"location": "Seattle", "timezone": "UTC", "verbose": True} + ) + self.assertEqual(valid_request_with_data.args, ["arg1", "arg2", 123]) + self.assertEqual( + valid_request_with_data.kwargs, + {"location": "Seattle", "timezone": "UTC", "verbose": True} + ) + + # Test with complex args and kwargs + complex_request = SkillApiRequestData( + args=[ + {"nested": "dict"}, + ["list", "of", "items"], + 42, + True, + None + ], + kwargs={ + "nested_dict": {"key": "value", "number": 123}, + "nested_list": [1, 2, 3], + "boolean": False, + "null_value": None + } + ) + self.assertIsInstance(complex_request.args[0], dict) + self.assertIsInstance(complex_request.args[1], list) + self.assertEqual(complex_request.kwargs["nested_dict"]["number"], 123) + + # Test with empty lists/dicts + empty_request = SkillApiRequestData(args=[], kwargs={}) + self.assertEqual(len(empty_request.args), 0) + self.assertEqual(len(empty_request.kwargs), 0) + + def test_skill_api_response_data(self): + from neon_data_models.models.api.messagebus import SkillApiResponseData + + # Test valid response with result + valid_response_with_result = SkillApiResponseData( + result="API Response matching advertised schema", + error=None + ) + self.assertEqual( + valid_response_with_result.result, + "API Response matching advertised schema" + ) + self.assertIsNone(valid_response_with_result.error) + + # Test valid response with error + valid_response_with_error = SkillApiResponseData( + result=None, + error="API Method error message" + ) + self.assertIsNone(valid_response_with_error.result) + self.assertEqual(valid_response_with_error.error, "API Method error message") + + # Test valid response with complex result + complex_result = { + "timestamp": 1640995200.0, + "timezone": "UTC", + "formatted_time": "2022-01-01 00:00:00", + "metadata": { + "source": "system_clock", + "accuracy": "high" + } + } + valid_response_complex = SkillApiResponseData(result=complex_result) + self.assertEqual(valid_response_complex.result, complex_result) + self.assertIsNone(valid_response_complex.error) + + # Test valid response with list result + list_result = ["item1", "item2", "item3"] + valid_response_list = SkillApiResponseData(result=list_result) + self.assertEqual(valid_response_list.result, list_result) + + # Test response with both result and error (should be allowed) + both_fields_response = SkillApiResponseData( + result="some result", + error="some error" + ) + self.assertEqual(both_fields_response.result, "some result") + self.assertEqual(both_fields_response.error, "some error") + + # Test default values (both None) + default_response = SkillApiResponseData() + self.assertIsNone(default_response.result) + self.assertIsNone(default_response.error) + + # Test with various data types as result + int_result = SkillApiResponseData(result=42) + float_result = SkillApiResponseData(result=3.14159) + bool_result = SkillApiResponseData(result=True) + + self.assertEqual(int_result.result, 42) + self.assertEqual(float_result.result, 3.14159) + self.assertTrue(bool_result.result) + + def test_neon_call_skill_api(self): + from neon_data_models.models.api.messagebus import NeonCallSkillApi, SkillApiRequestData + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid message with specific msg_type + request_data = SkillApiRequestData( + args=["test_location"], + kwargs={"format": "iso"} + ) + valid_message = NeonCallSkillApi( + msg_type="skill-date_time.neongeckocom.get_current_time", + data=request_data, + context={} + ) + self.assertIsInstance(valid_message, NeonCallSkillApi) + self.assertIsInstance(valid_message, BaseMessage) + self.assertEqual(valid_message.msg_type, "skill-date_time.neongeckocom.get_current_time") + self.assertEqual(valid_message.data.args, ["test_location"]) + self.assertEqual(valid_message.data.kwargs, {"format": "iso"}) + + # Test with default data + message_with_default = NeonCallSkillApi( + msg_type="test.skill.method", + context={} + ) + self.assertEqual(message_with_default.data.args, []) + self.assertEqual(message_with_default.data.kwargs, {}) + + # Test with empty request data + empty_request = SkillApiRequestData() + message_empty = NeonCallSkillApi( + msg_type="test.skill.empty", + data=empty_request, + context={} + ) + self.assertEqual(message_empty.data.args, []) + self.assertEqual(message_empty.data.kwargs, {}) + + # Test with complex arguments + complex_args = [ + {"nested": "object"}, + [1, 2, 3], + "simple_string", + 42, + True + ] + complex_kwargs = { + "config": {"setting": "value"}, + "options": ["opt1", "opt2"], + "enabled": True + } + complex_request = SkillApiRequestData(args=complex_args, kwargs=complex_kwargs) + complex_message = NeonCallSkillApi( + msg_type="skill.complex.method", + data=complex_request, + context={"source": "test"} + ) + self.assertEqual(complex_message.data.args[0]["nested"], "object") + self.assertEqual(complex_message.data.kwargs["config"]["setting"], "value") + + # Test missing required msg_type + with self.assertRaises(ValidationError): + NeonCallSkillApi(data=request_data, context={}) + + def test_neon_call_skill_api_response(self): + from neon_data_models.models.api.messagebus import NeonCallSkillApiResponse, SkillApiResponseData + from neon_data_models.models.base.messagebus import BaseMessage + + # Test valid response with successful result + response_data = SkillApiResponseData( + result={ + "current_timestamp": 1640995200.0, + "formatted_time": "2022-01-01 00:00:00 UTC" + }, + error=None + ) + valid_response = NeonCallSkillApiResponse( + msg_type="skill-date_time.neongeckocom.get_current_time.response", + data=response_data, + context={} + ) + self.assertIsInstance(valid_response, NeonCallSkillApiResponse) + self.assertIsInstance(valid_response, BaseMessage) + self.assertEqual(valid_response.msg_type, "skill-date_time.neongeckocom.get_current_time.response") + self.assertEqual(valid_response.data.result["current_timestamp"], 1640995200.0) + self.assertIsNone(valid_response.data.error) + + # Test response with error + error_response_data = SkillApiResponseData( + result=None, + error="Failed to get current time: Network error" + ) + error_response = NeonCallSkillApiResponse( + msg_type="skill-date_time.neongeckocom.get_current_time.response", + data=error_response_data, + context={} + ) + self.assertIsNone(error_response.data.result) + self.assertEqual(error_response.data.error, "Failed to get current time: Network error") + + # Test with simple result types + string_response = NeonCallSkillApiResponse( + msg_type="test.skill.string.response", + data=SkillApiResponseData(result="Simple string result"), + context={} + ) + self.assertEqual(string_response.data.result, "Simple string result") + + number_response = NeonCallSkillApiResponse( + msg_type="test.skill.number.response", + data=SkillApiResponseData(result=42), + context={} + ) + self.assertEqual(number_response.data.result, 42) + + list_response = NeonCallSkillApiResponse( + msg_type="test.skill.list.response", + data=SkillApiResponseData(result=["item1", "item2", "item3"]), + context={} + ) + self.assertEqual(len(list_response.data.result), 3) + self.assertEqual(list_response.data.result[1], "item2") + + # Test with both result and error (edge case) + mixed_response = NeonCallSkillApiResponse( + msg_type="test.skill.mixed.response", + data=SkillApiResponseData( + result="Partial result", + error="Warning: incomplete data" + ), + context={} + ) + self.assertEqual(mixed_response.data.result, "Partial result") + self.assertEqual(mixed_response.data.error, "Warning: incomplete data") + + # Test missing required msg_type + with self.assertRaises(ValidationError): + NeonCallSkillApiResponse(data=response_data, context={}) + + # Test missing required data + with self.assertRaises(ValidationError): + NeonCallSkillApiResponse( + msg_type="test.response", + context={} + ) + diff --git a/tests/models/api/test_mq.py b/tests/models/api/test_mq.py index 2981a04b..ef5fe514 100644 --- a/tests/models/api/test_mq.py +++ b/tests/models/api/test_mq.py @@ -445,10 +445,28 @@ def test_neon_mq_tts_response(self): # with self.assertRaises(ValidationError): # NeonMqTtsResponse(message_id="test_mid") + def test_neon_mq_get_skills_api(self): + from neon_data_models.models.api.mq.neon import NeonMqGetSkillsApi + # TODO + + def test_neon_mq_skills_api_response(self): + from neon_data_models.models.api.mq.neon import NeonMqSkillsApiResponse + # TODO + + def test_neon_mq_call_skill_api(self): + from neon_data_models.models.api.mq.neon import NeonMqCallSkillApi + # TODO + + def test_neon_mq_call_skill_api_response(self): + from neon_data_models.models.api.mq.neon import NeonMqCallSkillApiResponse + # TODO + def test_neon_api_message(self): from neon_data_models.models.api.mq.neon import NeonApiMessage, GetTtsData, GetSttData, GetResponseData from neon_data_models.models.api.mq.neon import NeonMqGetTts, NeonMqGetStt, NeonMqTextInput from neon_data_models.models.api.mq.neon import NeonMqSttResponse, NeonMqTtsResponse + from neon_data_models.models.api.mq.neon import NeonMqGetSkillsApi, NeonMqSkillsApiResponse + from neon_data_models.models.api.mq.neon import NeonMqCallSkillApi, NeonMqCallSkillApiResponse # Test TTS message tts_message = NeonApiMessage( @@ -501,6 +519,62 @@ def test_neon_api_message(self): ) self.assertIsInstance(tts_response, NeonMqTtsResponse) + # Test Get Skills API message + skills_api_message = NeonApiMessage( + msg_type="neon.skill_api.get", + data={}, + context={}, + message_id="test_mid" + ) + self.assertIsInstance(skills_api_message, NeonMqGetSkillsApi) + + # Test Skills API Response message + skills_api_response = NeonApiMessage( + msg_type="neon.skill_api.get.response", + data={ + "skill-date_time.neongeckocom": { + "get_current_time": { + "help": "Get the current timestamp", + "type": "skill-date_time.neongeckocom.get_current_time", + "request_schema": None, + "response_schema": None, + "signature": None + } + } + }, + context={}, + message_id="test_mid" + ) + self.assertIsInstance(skills_api_response, NeonMqSkillsApiResponse) + + # Test Call Skill API message + call_skill_api_message = NeonApiMessage( + msg_type="neon.skill_api.call", + data={ + "msg_type": "skill-date_time.neongeckocom.get_current_time", + "args": ["Seattle"], + "kwargs": {"format": "iso"} + }, + context={}, + message_id="test_mid" + ) + self.assertIsInstance(call_skill_api_message, NeonMqCallSkillApi) + + # Test Call Skill API Response message + call_skill_api_response = NeonApiMessage( + msg_type="neon.skill_api.response", + data={ + "result": { + "current_timestamp": 1640995200.0, + "formatted_time": "2022-01-01 00:00:00 UTC" + }, + "error": None + }, + context={}, + message_id="test_mid" + ) + self.assertIsInstance(call_skill_api_response, NeonMqCallSkillApiResponse) + # Test from_sio_message for STT sio_stt = { "requested_skill": "stt", From 985cc0d0befe1fb5bff18ccb451ae37aa4475c25 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Fri, 15 Aug 2025 15:54:54 +0000 Subject: [PATCH 22/36] Increment Version to 0.0.2a11 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index e27a31d3..5e010aa2 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a10" +__version__ = "0.0.2a11" From 90c205d3b227019119b1ae44d60c27f78db6064f Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Thu, 25 Sep 2025 09:30:55 -0700 Subject: [PATCH 23/36] Add `context` support to ChatbotsMqRequest object (#28) --- neon_data_models/models/api/mq/chatbots.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/neon_data_models/models/api/mq/chatbots.py b/neon_data_models/models/api/mq/chatbots.py index 8e4c55e7..c4e52d74 100644 --- a/neon_data_models/models/api/mq/chatbots.py +++ b/neon_data_models/models/api/mq/chatbots.py @@ -64,6 +64,8 @@ class ChatbotsMqRequest(KlatContext, MQContext): default=None, description="Explicitly defined recipient of the shout") bound_service: Optional[str] = Field( default=None, description="Service bound to the conversation") + context: Optional[dict] = Field( + default=None, deprecated=True, description="Extra proctor context") @classmethod def from_sio_message(cls, sio_message: dict) -> 'ChatbotsMqRequest': From b77e1a61ee5d1fad7005f4ebf5c8a3ae9827b586 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Thu, 25 Sep 2025 16:31:06 +0000 Subject: [PATCH 24/36] Increment Version to 0.0.2a12 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 5e010aa2..68b5ddf7 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a11" +__version__ = "0.0.2a12" From 30b072642e6a6cf86f8083d7d6b0807daeaa5bfc Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Tue, 7 Oct 2025 13:16:23 -0700 Subject: [PATCH 25/36] Implement Models for Klat SocketIO events (#11) * Implement Klat SocketIO data models * Refactor model names to match associated SIO event names * Annotate PromptData Define GetPromptData Socketio data structure * Annotate PromptData Define GetPromptData Socketio data structure * Add a validator to `NewMessage` to support deprecated `message_body` input referenced in Klat Observer * Annotate `NewMessage.promptState` as Optional * Refactor `id` to `_id` instead of aliasing * Override `_PromptData.model_dump` to dump by alias by default Add unit test case to validate PromptData serialization * override serialization instead of model_dump and increase test coverage to resolve PromptData serialization bug * Update `isAnnouncement`, `timeCreated`, and `messageID` to validate types observed on Klat SocketIO API * Refactor after rebase on dev * Update based on pyklatchat-server current implementation * Implement SIO message models for everything in pyklatchat-server * Address issues noted in klat-server implementation and format file * Fix missed refactor from `promptState` -> `prompt_state` * Add backwards-compat. validation * Address observed errors in deployment * Add backwards-compat. parsing to UserMessage * Fix `bound_service` default value handling * Add backwards-compat serialization * Update CcaiPromptCompleted to extend UserMessage * Troubleshooting CcaiPromptCompleted handling * Troubleshooting CcaiPromptCompleted parsing * More `CcaiPromptCompleted` `prompt_id` value troubleshooting * Include backwards-compat. key in CcaiPromptCompleted message * Add default "source" value * Add `sid` to `UserMessage` as observed in current deployment * Update `UserMessage` default values to match actual behavior * Clean up SIO models to simplify alias handling * Fix type annotation of `bound_service` to match current implementation * Troubleshooting `is_bot` alias checks in alpha * Fix `NewPromptMessage` structure and docstrings * Fix `NewPromptMessage` creation bug * Troubleshooting Proctor `bot` flag handling * Troubleshooting backwards-compat * Update userID vs nick handling * Troubleshooting message serialization * Tighten up user_id validation and handling * Troubleshooting exceptions * Refactor to be consistent with `user_id` and `user_uid` params * Troubleshooting * More troubleshooting user_uid * More user_uid troubleshooting * Implement minimal UserMessage test case and resolve raised exception * Troubleshooting backwards-compat * Clean up changes * Update NewCcaiPrompt to support returning completed prompts * Clean up changes * Add `user_uid` to STT/TTS requests to match currently emitted events * Clean up `PromptData` to use bool instead of strings to represent a bool * Enforce `username` or `user_id` must be defined in `UserMessage` and Implement Unit Tests * Remove unused CcaiPromptCompleted.request_id * Refactor test case to specify "SocketIO" --- neon_data_models/models/api/__init__.py | 1 + neon_data_models/models/api/klat/__init__.py | 27 + neon_data_models/models/api/klat/socketio.py | 561 +++++++++++++++++++ tests/models/api/test_klat.py | 536 ++++++++++++++++++ 4 files changed, 1125 insertions(+) create mode 100644 neon_data_models/models/api/klat/__init__.py create mode 100644 neon_data_models/models/api/klat/socketio.py create mode 100644 tests/models/api/test_klat.py diff --git a/neon_data_models/models/api/__init__.py b/neon_data_models/models/api/__init__.py index e0e9c2bb..bbc589a1 100644 --- a/neon_data_models/models/api/__init__.py +++ b/neon_data_models/models/api/__init__.py @@ -29,3 +29,4 @@ from neon_data_models.models.api.llm import * from neon_data_models.models.api.mq import * from neon_data_models.models.api.messagebus import * +from neon_data_models.models.api.klat import * \ No newline at end of file diff --git a/neon_data_models/models/api/klat/__init__.py b/neon_data_models/models/api/klat/__init__.py new file mode 100644 index 00000000..2bd33fb0 --- /dev/null +++ b/neon_data_models/models/api/klat/__init__.py @@ -0,0 +1,27 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2024 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +from neon_data_models.models.api.klat.socketio import * diff --git a/neon_data_models/models/api/klat/socketio.py b/neon_data_models/models/api/klat/socketio.py new file mode 100644 index 00000000..90bf175b --- /dev/null +++ b/neon_data_models/models/api/klat/socketio.py @@ -0,0 +1,561 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2025 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +import uuid + +from time import time +from typing import Optional, Dict, List, Literal, Union, Any +from datetime import datetime +from pydantic import ( + Field, + model_validator, + model_serializer, + ConfigDict, +) + +from neon_data_models.enum import CcaiState +from neon_data_models.models.base import BaseModel + + +class GetSttRequest(BaseModel): + cid: str = Field(description="Conversation ID associated with the request") + sid: str = Field( + description="Shout ID associated with the request", + alias="message_id" + ) + user_uid: str = Field( + description="User UUID associated with the request", + alias="user_id", + deprecated=True + ) + lang: str = Field( + description="BCP-47 Language code associated with audio", + default="en-us", + ) + + +class GetSttResponse(BaseModel): + transcript: str = Field(description="Transcribed text") + lang: str = Field( + description="BCP-47 Language code associated with `transcript`", + default="en-us", + ) + sid: str = Field( + description="Shout ID associated with the request", + alias="message_id" + ) + cid: str = Field(description="Conversation ID associated with the request") + context: Dict[str, Any] = Field(default={}, description="Optional context") + + @model_validator(mode="before") + @classmethod + def parse_context(cls, values): + """ + Parse out message_id, sid, and cid from context if not handled by the + Observer module. + """ + values.setdefault("sid", values.get("context", {}).get("sid")) + values.setdefault("cid", values.get("context", {}).get("cid")) + return values + + +class GetTtsRequest(BaseModel): + cid: str = Field(description="Conversation ID associated with the request") + sid: str = Field( + description="Shout ID associated with the request", + alias="message_id" + ) + user_uid: str = Field( + description="User UUID associated with the request", + alias="user_id", + deprecated=True + ) + lang: str = Field( + description="BCP-47 Language code associated with `text`", + default="en-us", + ) + + +class GetTtsResponse(BaseModel): + audio_data: str = Field( + description="B64-encoded WAV audio file object generated from `text`" + ) + lang: str = Field( + description="BCP-47 Language code associated with `audio_data`", + default="en-us", + ) + gender: Literal["male", "female", "undefined"] = Field( + description="Gender associated with generated audio", + default="undefined", + ) + sid: str = Field( + description="Shout ID associated with the request", + alias="message_id" + ) + cid: str = Field(description="Conversation ID associated with the request") + context: Dict[str, Any] = Field(default={}, description="Optional context") + + @model_validator(mode="before") + @classmethod + def parse_context(cls, values): + """ + Parse out sid, and cid from context if not handled by the + Observer module. + """ + values.setdefault("sid", values.get("context", {}).get("sid")) + values.setdefault("cid", values.get("context", {}).get("cid")) + return values + + def to_db_query(self) -> Dict[str, Any]: + return { + "shout_id": self.sid, + "audio_data": self.audio_data, + "lang": self.lang, + "gender": self.gender, + } + + def to_incoming_tts(self) -> Dict[str, Any]: + return { + "cid": self.cid, + "sid": self.sid, + "audio_data": self.audio_data, + "lang": self.lang, + "gender": self.gender, + } + + +class NewPromptMessage(BaseModel): + """ + Model representing a user message that relates to a CCAI prompt. + """ + cid: str = Field(description="Conversation ID associated with the prompt and this response") + user_id: str = Field( + description="User ID (nick + suffix) associated with this message", + alias="userID" + ) + user_uid: str = Field( + description="User UUID associated with this message" + ) + prompt_id: str = Field( + description="Unique ID for the prompt this message relates to", alias="promptID" + ) + prompt_state: CcaiState = Field( + description="CCAI state this response is associated with", + default=CcaiState.IDLE, + alias="promptState", + ) + message_text: str = Field( + description="Submind response content", + alias="messageText", + ) + context: Dict[str, Any] = Field( + description="Extra context for the prompt", default={} + ) + + # Allow creation by name and alias inputs for backwards-compat. + model_config = ConfigDict(validate_by_alias=True, validate_by_name=True) + + def model_dump(self, **kwargs): + """ + Override model_dump to include SIO fields for backwards compatibility + """ + + # For backwards-compat with Klat Client, include aliased keys in + # serialization. In the future, this should be configurable and + # eventually removed. + by_alias = {} + if "by_alias" not in kwargs: + by_alias = super().model_dump(by_alias=True, **kwargs) + + return {**super().model_dump(**kwargs), **by_alias} + + +class UserMessage(BaseModel): + sid: str = Field( + description="Shout ID associated with the message" + ) + cid: str = Field(description="Conversation ID associated with the message") + user_id: Optional[str] = Field( + default=None, + description="User ID (nick + suffix) associated with the user", + alias="userID", + ) + user_uid: Optional[str] = Field(default=None, description="User UUID") + username: Optional[str] = Field( + description="Username of the sender", + default=None, + alias="userDisplayName" + ) + prompt_id: Optional[str] = Field( + default=None, + description="Prompt ID this message is in response to", + alias="promptID", + ) + prompt_state: Optional[CcaiState] = Field( + default=None, + alias="promptState", + description="Associated CCAI state if `prompt_id` is defined", + ) + source: str = Field( + default="unknown", description="Service associated with the message" + ) + message_body: str = Field( + description="Message content (input string or audio filename)", + alias="messageText" + ) + replied_message: Optional[str] = Field( + default=None, + description="Message ID this message is a reply to", + alias="repliedMessage" + ) + is_bot: Literal["0", "1"] = Field( + default="0", + description="'1' if the message came from a bot, else '0'", + alias="bot", + ) + lang: str = Field(default="en", description="ISO 639-1 Language code") + attachments: List[str] = Field( + default=[], + description="List of string filenames attached to this message", + ) + context: dict = Field(default={}, description="Optional arbitrary context") + is_audio: bool = Field( + default=False, + description="True if message_body represents encoded WAV audio", + ) + message_tts: Dict[str, Dict[Literal["male", "female"], str]] = Field( + default={}, + alias="messageTTS", + description="TTS Audio formatted as {: {: " + "}}", + ) + is_announcement: bool = Field( + description="True if the message is a system announcement", + default=False, + ) + time_created: datetime = Field( + description="Unix timestamp (epoch seconds)", + alias="timeCreated" + ) + message_id: str = Field( + description="UUID for this message", + alias="messageID", + default_factory=lambda: uuid.uuid4().hex[:10], + ) # TODO: This is probably `sid` with a different name + bound_service: Optional[str] = Field( + default=None, + description="Service this message is targeting", + alias="service_name", + ) + + # Below are observed as used, but purpose is unclear or deprecated + no_save: bool = Field(default=False, deprecated=True) + title: str = Field(default="", deprecated=True) + routing_key: Optional[str] = Field(default=None, deprecated=True) + bot_type: Optional[Any] = Field(default=None, deprecated=True) + omit_reply: bool = Field(default=False, deprecated=True) + to_discussion: bool = Field(default=False, deprecated=True) + dom: Optional[str] = Field(default=None, deprecated=True) + test: bool = Field( + default=False, + description="True if this message is associated with testing", + deprecated=True + ) + + @model_validator(mode="before") + @classmethod + def validate_inputs(cls, values): + if values.get("isAnnouncement") and "is_announcement" not in values: + values["is_announcement"] = values.get("isAnnouncement") == 1 + if values.get("isAudio") and "is_audio" not in values: + values["is_audio"] = values.get("isAudio") == 1 + if values.get("userDisplayName") and values.get("nick") and values['nick'].startswith(values['userDisplayName']): + # Patch old behavior and ensure `user_id` is nick + suffix + values['userID'] = values.pop('nick') + return values + + @model_validator(mode="after") + def validate_user_params(self): + # Client appears to send a UID as a nick + if self.user_id == self.username and self.user_id is not None: + raise ValueError(f"user_id should be a nick + suffix, " + f"not nick ({self.user_id})") + if self.username is None: + if self.user_id is None: + raise ValueError("Either user_id or username must be defined") + self.username = self.user_id.split('-')[0] or "guest" + return self + + class Config: + use_enum_values = True + validate_default = True + # Allow creation by name and alias inputs for backwards-compat. + validate_by_alias=True + validate_by_name=True + + def to_db_query(self) -> Dict[str, Any]: + return { + "_id": self.message_id, + "cid": self.cid, + "user_id": self.user_uid, + "prompt_id": self.prompt_id, + "message_text": self.message_body, + "message_lang": self.lang, + "attachments": self.attachments, + "replied_message": self.replied_message, + "is_audio": self.is_audio, + "is_announcement": self.is_announcement, + "is_bot": self.is_bot, + "translations": {}, + "created_on": int(self.time_created.timestamp()), + } + + def to_new_prompt_message(self) -> NewPromptMessage: + return NewPromptMessage( + cid=self.cid, + user_id=self.user_id, + user_uid=self.user_uid, + prompt_id=self.prompt_id, + prompt_state=self.prompt_state, + context=self.context, + message_text=self.message_body + ) + + def model_dump(self, **kwargs): + """ + Override model_dump to include SIO fields for backwards compatibility + """ + + # For backwards-compat with Klat Client, include aliased keys in + # serialization. In the future, this should be configurable and + # eventually removed. + by_alias = {} + if "by_alias" not in kwargs: + by_alias = super().model_dump(by_alias=True, **kwargs) + return {**super().model_dump(**kwargs), **by_alias} + + +class NewCcaiPrompt(BaseModel): + prompt_text: str = Field( + description="Text of the prompt, without the '!PROMPT: ' prefix" + ) + cid: str = Field(description="Conversation ID associated with the prompt") + prompt_id: str = Field(description="Unique ID for the prompt") + created_on: int = Field( + descrtion="Epoch seconds of prompt creation", + default_factory=lambda: int(time()), + ) + context: Dict[str, Any] = Field( + description="Extra context for the prompt", default={} + ) + # Completed prompts sent to the client use below fields + winner: Optional[str] = Field(default=None, description="Winning response User ID") + participating_subminds: List[str] = Field( + default=[], + description="List of subminds by User ID that participated in this prompt", + ) + proposed_responses: Dict[str, str] = Field( + default={}, + description="Dict of participating submind User ID to proposed response", + ) + votes: Dict[str, str] = Field( + default={}, + description="Dict of participating submind User ID to vote", + ) + submind_discussion_history: List[Dict[str, str]] = Field( + default=[], + description="List of discussoion round dicts of submind User ID to opinion response", + ) + + @model_validator(mode="before") + @classmethod + def validate_inputs(cls, values): + # Handle an invalid input context as a valid empty dict + # for backwards compatibility + if values.get("context", {}) is None: + values["context"] = {} + if values.get("submind_opinions"): + values["submind_discussion_history"] = [values.pop("submind_opinions")] + return values + + def to_db_query(self) -> Dict[str, Any]: + return { + "_id": self.prompt_id, + "cid": self.cid, + "is_completed": "0", + "data": {"prompt_text": self.prompt_text}, + "created_on": self.created_on, + "context": self.context, + } + + +class CcaiPromptCompleted(UserMessage): + winner: str = Field( + default="", + description="Winning response text; empty in the event of an error", + ) + prompt_id: str = Field( + description="Prompt ID this message is in response to", + ) + sid: str = Field( + description="Shout ID associated with the request" + ) + conversation_context: Dict[str, Any] = Field( + description="Context of the conversation", default={} + ) + + @model_validator(mode="before") + @classmethod + def validate_inputs(cls, values): + values.setdefault( + "winner", values.get("context", {}).get("winner", "") + ) + if values.get("prompt_id") in (None, ""): + # TODO: Figure out where this is set to an invalid value + values.pop("prompt_id") + values.setdefault( + "prompt_id", + values.get("context", {}).get("prompt", {}).get("prompt_id"), + ) + assert values["prompt_id"] != "", ( + f"prompt_id must be defined: {values}" + ) + return values + + @model_validator(mode="after") + def validate_fields(self): + # Client appears to send a UID as a nick + if self.user_id == self.username: + # TODO: This is patching backwards-compat. + self.user_id = None + if self.username is None: + self.username = "guest" + return self + + def to_db_query(self) -> Dict[str, Any]: + return { + "prompt_id": self.prompt_id, + "prompt_context": self.context, + } + + def model_dump(self, **kwargs): + """ + Override model_dump to include SIO fields for backwards compatibility + """ + + # For backwards-compat with Klat Client, include aliased keys + by_alias = {"promptID": self.prompt_id} + return {**super().model_dump(**kwargs), **by_alias} + + +class GetPromptData(BaseModel): + nick: str = Field(description="Nickname of user requesting prompt data") + cid: str = Field(description="Conversation ID associated with the prompt") + limit: int = Field( + default=5, + description="Maximum number of prompts to return if `prompt_id` " + "is unset", + ) + prompt_id: Optional[str] = Field( + default=None, description="Optional prompt ID to get data for" + ) + + def to_db_query(self) -> Dict[str, Any]: + assert self.prompt_id is not None, "prompt_id must be defined" + return { + "cid": self.cid, + "limit": self.limit, + "prompt_ids": [self.prompt_id], + "fetch_user_data": True, + } + + +class PromptData(BaseModel): + class _PromptData(BaseModel): + id: str = Field(alias="_id", description="Unique ID for the prompt") + is_completed: bool = Field( + description="true if a response to the prompt has been determined" + ) + proposed_responses: Dict[str, str] = Field( + default={}, + description="Dict of participant name to proposed response", + ) + submind_opinions: Dict[str, str] = Field( + default={}, + description="Dict of participant name to opinion response", + ) + votes: Dict[str, str] = Field( + default={}, description="Dict of participant name to vote" + ) + participating_subminds: List[str] = Field( + default=[], + description="List of subminds that participated in this prompt", + ) + + @model_validator(mode="before") + @classmethod + def normalize_completed(cls, values): + if "is_completed" in values and isinstance(values["is_completed"], str): + values["is_completed"] = values["is_completed"] == "1" + return values + + @model_serializer + def alias_serialize(self): + # Alias to match existing MongoDB schema + return { + "_id": self.id, + "is_completed": "1" if self.is_completed else "0", + "proposed_responses": self.proposed_responses, + "submind_opinions": self.submind_opinions, + "votes": self.votes, + "participating_subminds": self.participating_subminds, + } + + data: Union[_PromptData, List[_PromptData]] = Field( + description="Prompt data" + ) + receiver: str = Field( + description="Nickname of user requesting prompt data" + ) + cid: str = Field(description="Conversation ID associated with the prompt") + request_id: str = Field( + default_factory=lambda: str(uuid.uuid4()), + description="Unique ID of the request to identify the response", + ) + + +__all__ = [ + GetSttRequest.__name__, + GetSttResponse.__name__, + GetTtsRequest.__name__, + GetTtsResponse.__name__, + UserMessage.__name__, + NewPromptMessage.__name__, + GetPromptData.__name__, + NewCcaiPrompt.__name__, + CcaiPromptCompleted.__name__, + PromptData.__name__, +] diff --git a/tests/models/api/test_klat.py b/tests/models/api/test_klat.py new file mode 100644 index 00000000..67b8792f --- /dev/null +++ b/tests/models/api/test_klat.py @@ -0,0 +1,536 @@ +# NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System +# All trademark and other rights reserved by their respective owners +# Copyright 2008-2024 Neongecko.com Inc. +# BSD-3 +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# 3. Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from this +# software without specific prior written permission. +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, +# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, +# OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING +# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +import base64 +from unittest import TestCase +from datetime import datetime +from pydantic import ValidationError + +from neon_data_models.enum import CcaiState + + +class TestKlatSocketIO(TestCase): + def test_get_stt_request(self): + """Test GetSttRequest model creation and validation""" + from neon_data_models.models.api.klat.socketio import GetSttRequest + + # Test with required fields + test_object = GetSttRequest( + cid="test_conversation_id", + sid="test_shout_id", + user_uid="test_user_id" + ) + self.assertIsInstance(test_object, GetSttRequest) + self.assertEqual(test_object.cid, "test_conversation_id") + self.assertEqual(test_object.sid, "test_shout_id") + self.assertEqual(test_object.user_uid, "test_user_id") + self.assertEqual(test_object.lang, "en-us") # Default value + + # Test with alias parameter + test_object = GetSttRequest( + cid="test_conversation_id", + message_id="test_message_id", + user_id="test_user_uid" + ) + self.assertEqual(test_object.sid, "test_message_id") + self.assertEqual(test_object.user_uid, "test_user_uid") + + # Test serialization/deserialization + dumped = test_object.model_dump() + recreated = GetSttRequest(**dumped) + self.assertEqual(test_object, recreated) + + def test_get_stt_response(self): + """Test GetSttResponse model creation and validation""" + from neon_data_models.models.api.klat.socketio import GetSttResponse + + # Test with required fields + test_object = GetSttResponse( + transcript="Hello world", + sid="test_shout_id", + cid="test_conversation_id" + ) + self.assertIsInstance(test_object, GetSttResponse) + self.assertEqual(test_object.transcript, "Hello world") + self.assertEqual(test_object.sid, "test_shout_id") + self.assertEqual(test_object.cid, "test_conversation_id") + self.assertEqual(test_object.lang, "en-us") # Default value + self.assertEqual(test_object.context, {}) # Default value + + # Test with context extraction + context = {"sid": "context_sid", "cid": "context_cid", "extra": "data"} + test_object = GetSttResponse( + transcript="Hello world", + context=context + ) + self.assertEqual(test_object.sid, "context_sid") + self.assertEqual(test_object.cid, "context_cid") + self.assertEqual(test_object.context["extra"], "data") + + # Test serialization/deserialization + dumped = test_object.model_dump() + recreated = GetSttResponse(**dumped) + self.assertEqual(test_object, recreated) + + def test_get_tts_request(self): + """Test GetTtsRequest model creation and validation""" + from neon_data_models.models.api.klat.socketio import GetTtsRequest + + # Test with required fields + test_object = GetTtsRequest( + cid="test_conversation_id", + sid="test_shout_id", + user_uid="test_user_uid", + ) + self.assertIsInstance(test_object, GetTtsRequest) + self.assertEqual(test_object.cid, "test_conversation_id") + self.assertEqual(test_object.sid, "test_shout_id") + self.assertEqual(test_object.user_uid, "test_user_uid") + self.assertEqual(test_object.lang, "en-us") # Default value + + # Test with alias parameters + test_object = GetTtsRequest( + cid="test_conversation_id", + message_id="test_message_id", + user_id="test_user_uid", + ) + self.assertEqual(test_object.sid, "test_message_id") + self.assertEqual(test_object.user_uid, "test_user_uid") + + # Test serialization/deserialization + dumped = test_object.model_dump() + recreated = GetTtsRequest(**dumped) + self.assertEqual(test_object, recreated) + + def test_get_tts_response(self): + """Test GetTtsResponse model creation and validation""" + from neon_data_models.models.api.klat.socketio import GetTtsResponse + + # Mock base64 audio data + mock_audio = base64.b64encode(b"mock audio data").decode() + + # Test with required fields + test_object = GetTtsResponse( + audio_data=mock_audio, + sid="test_shout_id", + cid="test_conversation_id" + ) + self.assertIsInstance(test_object, GetTtsResponse) + self.assertEqual(test_object.audio_data, mock_audio) + self.assertEqual(test_object.sid, "test_shout_id") + self.assertEqual(test_object.cid, "test_conversation_id") + self.assertEqual(test_object.lang, "en-us") # Default value + self.assertEqual(test_object.gender, "undefined") # Default value + self.assertEqual(test_object.context, {}) # Default value + + # Test with context extraction + context = {"sid": "context_sid", "cid": "context_cid"} + test_object = GetTtsResponse( + audio_data=mock_audio, + context=context + ) + self.assertEqual(test_object.sid, "context_sid") + self.assertEqual(test_object.cid, "context_cid") + + # Test to_db_query method + db_query = test_object.to_db_query() + self.assertEqual(db_query["shout_id"], test_object.sid) + self.assertEqual(db_query["audio_data"], test_object.audio_data) + self.assertEqual(db_query["lang"], test_object.lang) + self.assertEqual(db_query["gender"], test_object.gender) + + # Test to_incoming_tts method + incoming_tts = test_object.to_incoming_tts() + self.assertEqual(incoming_tts["cid"], test_object.cid) + self.assertEqual(incoming_tts["sid"], test_object.sid) + self.assertEqual(incoming_tts["audio_data"], test_object.audio_data) + self.assertEqual(incoming_tts["lang"], test_object.lang) + self.assertEqual(incoming_tts["gender"], test_object.gender) + + # Test serialization/deserialization + dumped = test_object.model_dump() + recreated = GetTtsResponse(**dumped) + self.assertEqual(test_object, recreated) + + def test_new_prompt_message(self): + """Test NewPromptMessage model creation and validation""" + from neon_data_models.models.api.klat.socketio import NewPromptMessage + + # Test with required fields + test_object = NewPromptMessage( + cid="test_conversation_id", + user_id="test_user-suffix", + user_uid="test_user_uuid", + prompt_id="test_prompt_id", + message_text="Hello world" + ) + self.assertIsInstance(test_object, NewPromptMessage) + self.assertEqual(test_object.cid, "test_conversation_id") + self.assertEqual(test_object.user_id, "test_user-suffix") + self.assertEqual(test_object.user_uid, "test_user_uuid") + self.assertEqual(test_object.prompt_id, "test_prompt_id") + self.assertEqual(test_object.message_text, "Hello world") + self.assertEqual(test_object.prompt_state, CcaiState.IDLE) # Default value + self.assertEqual(test_object.context, {}) # Default value + + # Test with alias parameters + test_object = NewPromptMessage( + cid="test_conversation_id", + userID="test_user-suffix", + user_uid="test_user_uuid", + promptID="test_prompt_id", + messageText="Hello world", + promptState=CcaiState.RESP + ) + self.assertEqual(test_object.user_id, "test_user-suffix") + self.assertEqual(test_object.prompt_id, "test_prompt_id") + self.assertEqual(test_object.message_text, "Hello world") + self.assertEqual(test_object.prompt_state, CcaiState.RESP) + + # Test model_dump with aliases + dumped = test_object.model_dump() + self.assertIn("user_id", dumped) + self.assertIn("userID", dumped) # Should include alias + self.assertEqual(dumped["user_id"], dumped["userID"]) + self.assertIn("prompt_id", dumped) + self.assertIn("promptID", dumped) # Should include alias + self.assertEqual(dumped["prompt_id"], dumped["promptID"]) + + # Test serialization/deserialization + recreated = NewPromptMessage(**dumped) + self.assertEqual(test_object, recreated) + + def test_user_message(self): + """Test UserMessage model creation and validation""" + from neon_data_models.models.api.klat.socketio import UserMessage + + # Test with required fields + current_time = datetime.now() + test_object = UserMessage( + sid="test_shout_id", + cid="test_conversation_id", + message_body="Hello world", + time_created=current_time, + user_id="guest-sid", + ) + self.assertIsInstance(test_object, UserMessage) + self.assertEqual(test_object.sid, "test_shout_id") + self.assertEqual(test_object.cid, "test_conversation_id") + self.assertEqual(test_object.message_body, "Hello world") + self.assertEqual(test_object.time_created, current_time) + self.assertEqual(test_object.user_id, "guest-sid") + self.assertEqual(test_object.username, "guest") # Parsed from user_id + self.assertEqual(test_object.source, "unknown") # Default value + self.assertEqual(test_object.is_bot, "0") # Default value + self.assertEqual(test_object.lang, "en") # Default value + + # Test with alias parameters and username derivation + test_object = UserMessage( + sid="test_shout_id", + cid="test_conversation_id", + userID="test_user-suffix", + messageText="Hello world", + time_created=current_time + ) + self.assertEqual(test_object.user_id, "test_user-suffix") + self.assertEqual(test_object.message_body, "Hello world") + # Username should be derived from user_id + self.assertEqual(test_object.username, "test_user") + + # Test with validation + try: + UserMessage( + sid="test_shout_id", + cid="test_conversation_id", + user_id="same_name", + userDisplayName="same_name", # Should raise error when same as user_id + message_body="Hello world", + time_created=current_time + ) + self.fail("Should have raised ValueError") + except ValueError as e: + self.assertIn("user_id should be a nick + suffix", str(e)) + + # Test isAnnouncement and isAudio aliases + test_object = UserMessage( + sid="test_shout_id", + cid="test_conversation_id", + message_body="Hello world", + time_created=current_time, + isAnnouncement=1, + isAudio=1, + username="test_user" + ) + self.assertTrue(test_object.is_announcement) + self.assertTrue(test_object.is_audio) + self.assertEqual(test_object.username, "test_user") + self.assertIsNone(test_object.user_id) # Derived from username + + # Test to_db_query method + db_query = test_object.to_db_query() + self.assertEqual(db_query["_id"], test_object.message_id) + self.assertEqual(db_query["cid"], test_object.cid) + self.assertEqual(db_query["message_text"], test_object.message_body) + self.assertEqual(db_query["is_audio"], test_object.is_audio) + self.assertEqual(db_query["is_announcement"], test_object.is_announcement) + self.assertEqual(db_query["created_on"], int(current_time.timestamp())) + + # Test to_new_prompt_message conversion + test_object = UserMessage( + sid="test_shout_id", + cid="test_conversation_id", + userID="test_user-suffix", + user_uid="test_uuid", + prompt_id="test_prompt_id", + prompt_state=CcaiState.VOTE, + message_body="Hello world", + time_created=current_time, + ) + prompt_message = test_object.to_new_prompt_message() + self.assertEqual(prompt_message.cid, test_object.cid) + self.assertEqual(prompt_message.user_id, test_object.user_id) + self.assertEqual(prompt_message.user_uid, test_object.user_uid) + self.assertEqual(prompt_message.prompt_id, test_object.prompt_id) + self.assertEqual(prompt_message.prompt_state, test_object.prompt_state) + self.assertEqual(prompt_message.message_text, test_object.message_body) + + def test_new_ccai_prompt(self): + """Test NewCcaiPrompt model creation and validation""" + from neon_data_models.models.api.klat.socketio import NewCcaiPrompt + + # Test with required fields + test_object = NewCcaiPrompt( + prompt_text="What is the meaning of life?", + cid="test_conversation_id", + prompt_id="test_prompt_id" + ) + self.assertIsInstance(test_object, NewCcaiPrompt) + self.assertEqual(test_object.prompt_text, "What is the meaning of life?") + self.assertEqual(test_object.cid, "test_conversation_id") + self.assertEqual(test_object.prompt_id, "test_prompt_id") + self.assertIsNotNone(test_object.created_on) + self.assertEqual(test_object.context, {}) # Default value + self.assertIsNone(test_object.winner) # Default value + self.assertEqual(test_object.participating_subminds, []) # Default value + self.assertEqual(test_object.proposed_responses, {}) # Default value + self.assertEqual(test_object.votes, {}) # Default value + self.assertEqual(test_object.submind_discussion_history, []) # Default value + + # Test with None context (should convert to empty dict) + test_object = NewCcaiPrompt( + prompt_text="What is the meaning of life?", + cid="test_conversation_id", + prompt_id="test_prompt_id", + context=None + ) + self.assertEqual(test_object.context, {}) + + # Test submind_opinions conversion to submind_discussion_history + submind_opinions = {"submind1": "opinion1", "submind2": "opinion2"} + test_object = NewCcaiPrompt( + prompt_text="What is the meaning of life?", + cid="test_conversation_id", + prompt_id="test_prompt_id", + submind_opinions=submind_opinions + ) + self.assertEqual(len(test_object.submind_discussion_history), 1) + self.assertEqual(test_object.submind_discussion_history[0], submind_opinions) + + # Test to_db_query method + db_query = test_object.to_db_query() + self.assertEqual(db_query["_id"], test_object.prompt_id) + self.assertEqual(db_query["cid"], test_object.cid) + self.assertEqual(db_query["is_completed"], "0") + self.assertEqual(db_query["data"]["prompt_text"], test_object.prompt_text) + self.assertEqual(db_query["created_on"], test_object.created_on) + self.assertEqual(db_query["context"], test_object.context) + + def test_ccai_prompt_completed(self): + """Test CcaiPromptCompleted model creation and validation""" + from neon_data_models.models.api.klat.socketio import CcaiPromptCompleted + + # Prepare a context with required fields + current_time = datetime.now() + context = { + "winner": "The meaning of life is 42", + "prompt": {"prompt_id": "test_prompt_id"} + } + + # Test with required fields + test_object = CcaiPromptCompleted( + sid="test_shout_id", + cid="test_conversation_id", + message_body="The meaning of life is 42", + time_created=current_time, + prompt_id="test_prompt_id", + user_id="proctor-sid", + context=context + ) + self.assertIsInstance(test_object, CcaiPromptCompleted) + self.assertEqual(test_object.sid, "test_shout_id") + self.assertEqual(test_object.cid, "test_conversation_id") + self.assertEqual(test_object.message_body, "The meaning of life is 42") + self.assertEqual(test_object.time_created, current_time) + self.assertEqual(test_object.prompt_id, "test_prompt_id") + self.assertEqual(test_object.winner, "The meaning of life is 42") + self.assertEqual(test_object.user_id, "proctor-sid") + self.assertEqual(test_object.username, "proctor") # Parsed from user_id + + # Test with username validation + with self.assertRaises(ValidationError): + test_object = CcaiPromptCompleted( + sid="test_shout_id", + cid="test_conversation_id", + user_id="same_name", + username="same_name", + message_body="Hello world", + time_created=current_time, + prompt_id="test_prompt_id" + ) + + # Test missing prompt_id handling + with self.assertRaises(ValidationError): + CcaiPromptCompleted( + sid="test_shout_id", + cid="test_conversation_id", + message_body="Hello world", + time_created=current_time, + prompt_id="" # Empty string should raise assertion + ) + + # Test to_db_query method + db_query = test_object.to_db_query() + self.assertEqual(db_query["prompt_id"], test_object.prompt_id) + self.assertEqual(db_query["prompt_context"], test_object.context) + + # Test model_dump with aliases + dumped = test_object.model_dump() + self.assertIn("prompt_id", dumped) + self.assertIn("promptID", dumped) # Should include alias + self.assertEqual(dumped["prompt_id"], dumped["promptID"]) + + def test_get_prompt_data(self): + """Test GetPromptData model creation and validation""" + from neon_data_models.models.api.klat.socketio import GetPromptData + + # Test with required fields + test_object = GetPromptData( + nick="test_user", + cid="test_conversation_id", + prompt_id="test_prompt_id" + ) + self.assertIsInstance(test_object, GetPromptData) + self.assertEqual(test_object.nick, "test_user") + self.assertEqual(test_object.cid, "test_conversation_id") + self.assertEqual(test_object.prompt_id, "test_prompt_id") + self.assertEqual(test_object.limit, 5) # Default value + + # Test with custom limit + test_object = GetPromptData( + nick="test_user", + cid="test_conversation_id", + prompt_id="test_prompt_id", + limit=10 + ) + self.assertEqual(test_object.limit, 10) + + # Test to_db_query method + db_query = test_object.to_db_query() + self.assertEqual(db_query["cid"], test_object.cid) + self.assertEqual(db_query["limit"], test_object.limit) + self.assertEqual(db_query["prompt_ids"], [test_object.prompt_id]) + self.assertTrue(db_query["fetch_user_data"]) + + # Test with missing prompt_id + test_object = GetPromptData( + nick="test_user", + cid="test_conversation_id" + ) + self.assertIsNone(test_object.prompt_id) + # to_db_query should raise assertion error when prompt_id is None + with self.assertRaises(AssertionError): + test_object.to_db_query() + + def test_prompt_data(self): + """Test PromptData model creation and validation""" + from neon_data_models.models.api.klat.socketio import PromptData + + # Test _PromptData nested class + test_inner_object = PromptData._PromptData( + _id="test_id", + is_completed='1' # String value should be converted to boolean + ) + self.assertIsInstance(test_inner_object, PromptData._PromptData) + self.assertEqual(test_inner_object.id, "test_id") + self.assertTrue(test_inner_object.is_completed) + self.assertEqual(test_inner_object.proposed_responses, {}) # Default value + self.assertEqual(test_inner_object.submind_opinions, {}) # Default value + self.assertEqual(test_inner_object.votes, {}) # Default value + self.assertEqual(test_inner_object.participating_subminds, []) # Default value + + # Test serialization of _PromptData + serialized = test_inner_object.model_dump() + self.assertEqual(serialized["_id"], "test_id") + self.assertTrue(serialized["is_completed"]) + + # Test with custom serializer to match MongoDB schema + serialized = test_inner_object.alias_serialize() + self.assertEqual(serialized["_id"], "test_id") + self.assertEqual(serialized["is_completed"], "1") # Should be string "1" + + # Test full PromptData with single _PromptData object + valid_prompt_data = PromptData( + data=test_inner_object, + receiver="test_user", + cid="test_conversation_id" + ) + self.assertIsInstance(valid_prompt_data, PromptData) + self.assertEqual(valid_prompt_data.data, test_inner_object) + self.assertEqual(valid_prompt_data.receiver, "test_user") + self.assertEqual(valid_prompt_data.cid, "test_conversation_id") + self.assertIsNotNone(valid_prompt_data.request_id) # Should have auto-generated UUID + + # Test full PromptData with list of _PromptData objects + test_inner_object2 = PromptData._PromptData( + _id="test_id2", + is_completed='0' + ) + valid_prompt_data = PromptData( + data=[test_inner_object, test_inner_object2], + receiver="test_user", + cid="test_conversation_id" + ) + self.assertIsInstance(valid_prompt_data.data, list) + self.assertEqual(len(valid_prompt_data.data), 2) + self.assertEqual(valid_prompt_data.data[0], test_inner_object) + self.assertEqual(valid_prompt_data.data[1], test_inner_object2) + + # Test serialization/deserialization + dumped = valid_prompt_data.model_dump() + recreated = PromptData(**dumped) + self.assertEqual(valid_prompt_data.receiver, recreated.receiver) + self.assertEqual(valid_prompt_data.cid, recreated.cid) + self.assertEqual(valid_prompt_data.request_id, recreated.request_id) From e3f1f1e9bfb7747c98a1cff3ccda7037463af536 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Tue, 7 Oct 2025 20:16:35 +0000 Subject: [PATCH 26/36] Increment Version to 0.0.2a13 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 68b5ddf7..cb6eea18 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a12" +__version__ = "0.0.2a13" From e1fe0eb88c299dd2d9ec4e31d93e544b0d8465cb Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Tue, 7 Oct 2025 15:10:14 -0700 Subject: [PATCH 27/36] Refactor UserMessage SIO model (#29) * Alias `messageID` to `sid` instead of redundant `message_id` * Update unit test to account for sid refactor * Update `UserMessage` model --- neon_data_models/models/api/klat/socketio.py | 10 +++------- tests/models/api/test_klat.py | 2 +- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/neon_data_models/models/api/klat/socketio.py b/neon_data_models/models/api/klat/socketio.py index 90bf175b..70794aaf 100644 --- a/neon_data_models/models/api/klat/socketio.py +++ b/neon_data_models/models/api/klat/socketio.py @@ -195,7 +195,8 @@ def model_dump(self, **kwargs): class UserMessage(BaseModel): sid: str = Field( - description="Shout ID associated with the message" + description="Shout ID associated with the message", + alias="messageID" ) cid: str = Field(description="Conversation ID associated with the message") user_id: Optional[str] = Field( @@ -260,11 +261,6 @@ class UserMessage(BaseModel): description="Unix timestamp (epoch seconds)", alias="timeCreated" ) - message_id: str = Field( - description="UUID for this message", - alias="messageID", - default_factory=lambda: uuid.uuid4().hex[:10], - ) # TODO: This is probably `sid` with a different name bound_service: Optional[str] = Field( default=None, description="Service this message is targeting", @@ -318,7 +314,7 @@ class Config: def to_db_query(self) -> Dict[str, Any]: return { - "_id": self.message_id, + "_id": self.sid, "cid": self.cid, "user_id": self.user_uid, "prompt_id": self.prompt_id, diff --git a/tests/models/api/test_klat.py b/tests/models/api/test_klat.py index 67b8792f..fecff9d9 100644 --- a/tests/models/api/test_klat.py +++ b/tests/models/api/test_klat.py @@ -291,7 +291,7 @@ def test_user_message(self): # Test to_db_query method db_query = test_object.to_db_query() - self.assertEqual(db_query["_id"], test_object.message_id) + self.assertEqual(db_query["_id"], test_object.sid) self.assertEqual(db_query["cid"], test_object.cid) self.assertEqual(db_query["message_text"], test_object.message_body) self.assertEqual(db_query["is_audio"], test_object.is_audio) From e973ab9bf333b5bd1735b536ee15be42a9b296f6 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Tue, 7 Oct 2025 22:10:26 +0000 Subject: [PATCH 28/36] Increment Version to 0.0.2a14 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index cb6eea18..66fbf914 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a13" +__version__ = "0.0.2a14" From 8abdfa8617ef8a60c7ee682ec1e5d41e4b933f8d Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Mon, 3 Nov 2025 18:08:53 -0800 Subject: [PATCH 29/36] Refactor Chatbots messaging to match Klat (#30) * Update SIO message handling to match SIO models Update param names/aliases to match username/user_id/user_uid spec * Troubleshooting backwards-compat with `validation_alias` * Update MQ tests to remove deprecated `nick` field references * Update to address test failures * Fix timestamp serialization * Add validator check to ensure `username` and `user_id` are valid * Troubleshooting observed errors * Fix `message_text` handling in ChatbotsMqNewPrompt * Resolve observed errors * Update username/user_id handling and test cases * Refactor `NewPrompt` message * Fix up NewPrompt test cases * Clean up changes --- neon_data_models/models/api/klat/socketio.py | 4 +- neon_data_models/models/api/mq/chatbots.py | 578 ++++++++++++------- tests/models/api/test_klat.py | 245 +++++--- tests/models/api/test_mq.py | 192 +----- 4 files changed, 562 insertions(+), 457 deletions(-) diff --git a/neon_data_models/models/api/klat/socketio.py b/neon_data_models/models/api/klat/socketio.py index 70794aaf..52363de4 100644 --- a/neon_data_models/models/api/klat/socketio.py +++ b/neon_data_models/models/api/klat/socketio.py @@ -288,7 +288,9 @@ def validate_inputs(cls, values): values["is_announcement"] = values.get("isAnnouncement") == 1 if values.get("isAudio") and "is_audio" not in values: values["is_audio"] = values.get("isAudio") == 1 - if values.get("userDisplayName") and values.get("nick") and values['nick'].startswith(values['userDisplayName']): + if values.get("userDisplayName") and values.get("nick") and \ + values['nick'].startswith(values['userDisplayName']) and \ + values['nick'] != values['userDisplayName']: # Patch old behavior and ensure `user_id` is nick + suffix values['userID'] = values.pop('nick') return values diff --git a/neon_data_models/models/api/mq/chatbots.py b/neon_data_models/models/api/mq/chatbots.py index c4e52d74..52a0713f 100644 --- a/neon_data_models/models/api/mq/chatbots.py +++ b/neon_data_models/models/api/mq/chatbots.py @@ -26,7 +26,7 @@ from typing import Any, Dict, Literal, Optional, List, Union from datetime import datetime, timezone -from pydantic import Field, model_validator +from pydantic import Field, model_validator, AliasChoices from neon_data_models.enum import SubmindStatus, CcaiState, CcaiControl from neon_data_models.types import BotType @@ -40,48 +40,69 @@ class ChatbotsMqRequest(KlatContext, MQContext): """ Defines a request from Klat to the Chatbots service. """ - username: str = Field(alias="nick", - description="Username (or 'nick') of the sender") + + username: str = Field( + alias="nick", description="Username (or 'nick') of the sender" + ) cid: str = Field(description="Conversation ID associated with the shout") - message_text: str = Field(alias="messageText", - description="Text content of the shout") + message_text: str = Field( + alias="messageText", description="Text content of the shout" + ) from_bot: bool = Field( default=False, - description="True if the shout is from a bot, False if from a user") + description="True if the shout is from a bot, False if from a user", + ) prompt_id: Optional[str] = Field( default=None, - description="ID of the CCAI prompt associated with the shout") + description="ID of the CCAI prompt associated with the shout", + ) prompt_state: Optional[CcaiState] = Field( - default=None, deprecated=True, - description="State of the CCAI conversation associated with the shout") + default=None, + deprecated=True, + description="State of the CCAI conversation associated with the shout", + ) time_created: datetime = Field( - default= datetime.now(tz=timezone.utc), - description="Timestamp when the shout was created") + default=datetime.now(tz=timezone.utc), + description="Timestamp when the shout was created", + ) requested_participants: Optional[List[str]] = Field( - default=None, alias="participating_subminds", - description="List of CCAI participants requested to handle the shout") + default=None, + alias="participating_subminds", + description="List of CCAI participants requested to handle the shout", + ) recipient: Optional[str] = Field( - default=None, description="Explicitly defined recipient of the shout") + default=None, description="Explicitly defined recipient of the shout" + ) bound_service: Optional[str] = Field( - default=None, description="Service bound to the conversation") - context: Optional[dict] = Field( - default=None, deprecated=True, description="Extra proctor context") - + default=None, description="Service bound to the conversation" + ) + context: dict = Field( + default={}, deprecated=True, description="Extra proctor context" + ) + @classmethod - def from_sio_message(cls, sio_message: dict) -> 'ChatbotsMqRequest': + def from_sio_message(cls, sio_message: dict) -> "ChatbotsMqRequest": + # Parse incoming message `sid`, `cid`, and `title` from SIO Message klat_context = KlatContext(**sio_message) - mq_context = MQContext(**sio_message) + # This is the first MQ message; define new context + mq_context = MQContext() return ChatbotsMqRequest( **klat_context.model_dump(exclude_none=True), **mq_context.model_dump(exclude_none=True), - username=sio_message.get("userDisplayName") or \ - sio_message.get("userID"), - message_text=sio_message["messageText"], - from_bot=sio_message.get("bot") == 1, - prompt_id = sio_message.get("promptID"), - prompt_state=sio_message.get("promptState"), - time_created=sio_message["timeCreated"], - recipient=sio_message.get("recipient"), + username=sio_message.get("username") + or sio_message.get("userDisplayName"), + message_text=sio_message.get("message_body") + or sio_message.get("messageText"), + from_bot=sio_message.get("is_bot", sio_message.get("bot")) == 1, + prompt_id=sio_message.get("prompt_id") + or sio_message.get("promptID"), + prompt_state=sio_message.get("prompt_state") + or sio_message.get("promptState"), + time_created=sio_message.get("time_created") + or sio_message.get("timeCreated"), + recipient=sio_message.get( + "recipient" + ), bound_service=sio_message.get("bound_service"), ) @@ -90,11 +111,11 @@ def model_dump(self, **kwargs): Override model_dump to include SIO fields for backwards compatibility """ - # For backwards-compat with Klat Server, include aliased keys in + # For backwards-compat with Klat Server, include aliased keys in # serialization. In the future, this should be configurable and # eventually removed. by_alias = {} - if 'by_alias' not in kwargs: + if "by_alias" not in kwargs: by_alias = super().model_dump(by_alias=True, **kwargs) # Add parameters for backwards-compat. @@ -107,171 +128,260 @@ class ChatbotsMqSubmindResponse(KlatContext, MQContext): """ Defines a chatbot response to a request. """ - user_id: str = Field(alias='userID', - description="Unique UID of the sender") - username: Optional[str] = Field(default=None, - alias="userDisplayName", - description="Username of the sender") - message_text: str = Field(alias="messageText", - description="Text content of the shout") + + user_id: str = Field( + alias="userID", description="Unique UID of the sender" + ) + username: Optional[str] = Field( + default=None, + alias="nick", + description="Username of the sender", + ) + message_text: str = Field( + alias=AliasChoices("messageText", "shout"), + description="Text content of the shout", + ) sid: str = Field(default="", alias="messageID", description="Shout ID") replied_message: Optional[str] = Field( - default=None, alias="repliedMessage", - description="ID of the shout being replied to") - bot: Literal["0", "1"] = Field(default='0', alias='is_bot', - description="1 if the shout is from a bot") + default=None, + alias=AliasChoices("repliedMessage", "responded_shout"), + description="ID of the shout being replied to", + ) + bot: Literal["0", "1"] = Field( + default="0", alias="is_bot", description="1 if the shout is from a bot" + ) prompt_id: Optional[str] = Field( - default=None, alias="promptID", - description="ID of the CCAI prompt associated with the shout") + default=None, + alias="promptID", + description="ID of the CCAI prompt associated with the shout", + ) is_announcement: bool = Field( - default=False, alias="isAnnouncement", - description="True if the shout is an announcement") + default=False, + alias="isAnnouncement", + description="True if the shout is an announcement", + ) time_created: datetime = Field( - default= datetime.now(tz=timezone.utc), alias="timeCreated", - description="Timestamp when the shout was created") + default=datetime.now(tz=timezone.utc), + alias=AliasChoices("timeCreated", "time", "created_on"), + description="Timestamp when the shout was created", + ) source: str = Field( default="klat_observer", - description="Name of the service originating the shout") - bot_type: Optional[BotType] = Field(default=None, deprecated=True, - description="Type of submind sending the shout") - + description="Name of the service originating the shout", + ) + bot_type: Optional[BotType] = Field( + default=None, + deprecated=True, + description="Type of submind sending the shout", + ) + # Below are deprecated fields for backwards-compat. service_name: Any = Field(default=None, deprecated=True) - context: Optional[dict] = Field( - default=None, deprecated=True, - description="Context used for Klat Server backwards-compat.") - dom: Any = Field(default=None, deprecated=True, - description="Domain of this conversation") + context: dict = Field( + default={}, + deprecated=True, + description="Context used for Klat Server backwards-compat.", + ) + dom: Any = Field( + default=None, + deprecated=True, + description="Domain of this conversation", + ) omit_reply: bool = Field( - default=True, deprecated=True, - description="If true, the Proctor will ignore this message") - no_save: bool = Field(default=False, deprecated=True, - description="If true, this message will be ignored") + default=True, + deprecated=True, + description="If true, the Proctor will ignore this message", + ) + no_save: bool = Field( + default=False, + deprecated=True, + description="If true, this message will be ignored", + ) to_discussion: bool = Field(default=False, deprecated=True) prompt_state: CcaiState = Field( - default=CcaiState.IDLE, deprecated=True, alias="promptState", - description="State of the CCAI conversation associated with the shout") - - @model_validator(mode='after') + default=CcaiState.IDLE, + deprecated=True, + alias=AliasChoices("promptState", "conversation_state"), + description="State of the CCAI conversation associated with the shout", + ) + + @model_validator(mode="after") def set_username_from_user_id(self): if self.username is None and self.user_id: - self.username = self.user_id.rsplit('-', 1)[0] + self.username = self.user_id.rsplit("-", 1)[0] + if self.username == self.user_id: + raise ValueError( + f"username cannot be the same as user_id: {self.username}" + ) return self - @model_validator(mode='before') + @model_validator(mode="before") @classmethod def validate_inputs(cls, values): if isinstance(values, dict): - # Some additional aliases for backwards-compat. - if "nick" in values: - values.setdefault("userID", values.get("nick")) - - if "shout" in values: - values.setdefault("messageText", values.get("shout")) - - if "responded_shout" in values: - values.setdefault("repliedMessage", - values.get("responded_shout")) - - if "time" in values: - values.setdefault("timeCreated", values.get("time")) - - if "created_on" in values: - values.setdefault("timeCreated", values.get("created_on")) - if "sid" in values and values["sid"] is None: values.pop("sid") - if "conversation_state" in values: - values.setdefault("promptState", - values.get("conversation_state")) - return values def model_dump(self, **kwargs): - # For backwards-compat with Klat Server, include aliased keys in + # For backwards-compat with Klat Server, include aliased keys in # serialization. In the future, this should be configurable and # eventually removed. by_alias = {} - if 'by_alias' not in kwargs: + if "by_alias" not in kwargs: by_alias = super().model_dump(by_alias=True, **kwargs) - by_alias['isAnnouncement'] = '1' if self.is_announcement else '0' - by_alias['nick'] = self.user_id - by_alias['responded_shout'] = self.replied_message - by_alias['shout'] = self.message_text - by_alias['time'] = self.time_created.timestamp() - by_alias['promptState'] = self.prompt_state.value - by_alias['created_on'] = self.time_created.timestamp() - + # Add all aliases for fields using AliasChoices + # username: AliasChoices("userDisplayName", "nick") + by_alias["userDisplayName"] = self.username + by_alias["nick"] = self.username + + # message_text: AliasChoices("messageText", "shout") + by_alias["messageText"] = self.message_text + by_alias["shout"] = self.message_text + + # replied_message: AliasChoices("repliedMessage", "responded_shout") + by_alias["repliedMessage"] = self.replied_message + by_alias["responded_shout"] = self.replied_message + + # prompt_id: alias="promptID" + by_alias["promptID"] = self.prompt_id + + # is_announcement: alias="isAnnouncement" + by_alias["isAnnouncement"] = "1" if self.is_announcement else "0" + + # time_created: AliasChoices("timeCreated", "time", "created_on") + if self.time_created: + by_alias["timeCreated"] = self.time_created.timestamp() + by_alias["time"] = self.time_created.timestamp() + by_alias["created_on"] = self.time_created.timestamp() + + # prompt_state: AliasChoices("promptState", "conversation_state") + if self.prompt_state: + by_alias["promptState"] = self.prompt_state.value + by_alias["conversation_state"] = self.prompt_state.value + return {**super().model_dump(**kwargs), **by_alias} class PromptCompletedContext(BaseModel): prompt: Optional[ChatbotsMqRequest] = Field( - default=None, description="Original request containing the prompt") + default=None, description="Original request containing the prompt" + ) prompt_text: str = Field(description="The string prompt that is completed") - available_subminds: List[str] = Field( # Seems to always match `participating_subminds` - default=[], description="List of subminds available to participate") + available_subminds: List[str] = ( + Field( # Seems to always match `participating_subminds` + default=[], description="List of subminds available to participate" + ) + ) participating_subminds: List[str] = Field( - default=[], description="List of subminds participating in the prompt") + default=[], description="List of subminds participating in the prompt" + ) proposed_responses: Dict[str, str] = Field( - default={}, description="Dict of nick to proposal") - + default={}, description="Dict of nick to proposal" + ) + # In the future, there will be a list of these for multi-round discussion + submind_discussion_history: Dict[str, List[str]] = Field( + default=[], + description="List of dict of discussion rounds (dict of nick to shout)", + ) submind_opinions: Dict[str, str] = Field( - default={}, description="Dict of nick to discussion") + default={}, description="Dict of nick to discussion" + ) - votes: Dict[str, str] = Field(default={}, - description="Dict of nick to vote") + votes: Dict[str, str] = Field( + default={}, description="Dict of nick to vote" + ) votes_per_submind: Dict[str, List[str]] = Field( - default={}, description="Dict of nick to list of received votes") - winner: str = Field(default="", description="Selected winner") + default={}, + description="Dict of nick to list of received votes (nicks)", + ) + winner: str = Field(default="", description="nick of selected winner") # Below are deprecated is_active: bool = Field( # Seems to report active all the time - default=False, deprecated=True, - description="True if a response has not yet been chosen") + default=False, + deprecated=True, + description="True if a response has not yet been chosen", + ) state: Optional[CcaiState] = Field( - default=CcaiState.PICK, deprecated=True, - description="State of the CCAI conversation (always PICK)") + default=CcaiState.PICK, + deprecated=True, + description="State of the CCAI conversation (always PICK)", + ) + + @model_validator(mode="before") + @classmethod + def validate_discussion_history(cls, values): + if ( + "submind_discussion_history" not in values + and "submind_opinions" in values + ): + values["submind_discussion_history"] = { + k: [v] for k, v in values["submind_opinions"].items() + } + return values class ChatbotsMqSavePrompt(ChatbotsMqSubmindResponse): context: PromptCompletedContext = Field( alias="conversation_context", - description="Definition of the completed discussion") - + description="Definition of the completed discussion", + ) + def model_dump(self, **kwargs): return ChatbotsMqSubmindResponse.model_dump(self, **kwargs) -class ChatbotsMqNewPrompt(ChatbotsMqSubmindResponse): +class ChatbotsMqNewPrompt(KlatContext, MQContext): prompt_id: str = Field( description="ID of the CCAI prompt associated with the shout" ) - user_id: Optional[str] = Field(default=None, alias="nick", - description="User ID of the proctor") prompt_text: str = Field(description="The new prompt being discussed") prompt_state: CcaiState = Field( - default=CcaiState.IDLE, deprecated=True, - description="Implemented for backwards-compat. New Prompt always IDLE") + default=CcaiState.IDLE, + deprecated=True, + description="Implemented for backwards-compat. New Prompt always IDLE", + ) + user_id: Optional[str] = Field( + default=None, + alias="userID", + validation_alias="userID", + description="User ID of the proctor", + ) + sid: str = Field(default="", alias="messageID", description="Shout ID") + username: Optional[str] = Field( + default=None, + alias="nick", + description="Username of the sender", + ) + time_created: datetime = Field( + default=datetime.now(tz=timezone.utc), + alias=AliasChoices("timeCreated", "time", "created_on"), + description="Timestamp when the shout was created", + ) + source: str = Field( + default="klat_observer", + description="Name of the service originating the shout", + ) + bot_type: Optional[BotType] = Field( + default=None, + deprecated=True, + description="Type of submind sending the shout", + ) discussion_rounds: int = Field( - default=2, - description="Number of discussion rounds per cycle for this prompt") - context: Optional[dict] = Field(default=None, deprecated=True, - alias="conversation_context") - - @model_validator(mode='before') - @classmethod - def validate_context(cls, values): - values.setdefault("message_text", values.get("shout", "")) - values.setdefault("user_id", values.get("nick")) - return values - - def model_dump(self, **kwargs): - return ChatbotsMqSubmindResponse.model_dump(self, **kwargs) + default=2, + description="Number of discussion rounds per cycle for this prompt", + ) + context: dict = Field( + default={}, + deprecated=True, + description="Conversation Context used by Klat Server", + alias="conversation_context", + ) class ChatbotsMqResponse: @@ -279,14 +389,20 @@ class ChatbotsMqResponse: Type adapter for validating an arbitrary MQ message. This will always return an instance that extends `BaseMessage` and `MQContext`. """ + @classmethod - def __new__(cls, *_, **kwargs) -> Union[ChatbotsMqSavePrompt, - ChatbotsMqNewPrompt, - ChatbotsMqSubmindResponse]: - message_text = kwargs.get("message_text") or kwargs.get("messageText") \ + def __new__( + cls, *_, **kwargs + ) -> Union[ + ChatbotsMqSavePrompt, ChatbotsMqNewPrompt, ChatbotsMqSubmindResponse + ]: + message_text = ( + kwargs.get("message_text") + or kwargs.get("messageText") or kwargs.get("shout") - kwargs['message_text'] = message_text - + ) + kwargs["message_text"] = message_text + if message_text == CcaiControl.SAVE_PROMPT_RESULTS.value: return ChatbotsMqSavePrompt(**kwargs) elif message_text == CcaiControl.CREATE_PROMPT.value: @@ -295,69 +411,86 @@ def __new__(cls, *_, **kwargs) -> Union[ChatbotsMqSavePrompt, return ChatbotsMqSubmindResponse(**kwargs) - - class ChatbotsMqSubmindsState(MQContext): class SubmindState(BaseModel): - submind_id: str = Field(description="Connected submind's user_id") + submind_id: str = Field( + description="Connected submind's ID (nickname + suffix)" + ) status: SubmindStatus = Field( - description="Subminds's status in a particular conversation") + description="Subminds's status in a particular conversation" + ) subminds_per_cid: Dict[str, List[SubmindState]] = Field( - description="List of submind participants per conversation ID") + description="List of submind participants per conversation ID" + ) connected_subminds: Dict[str, ConnectedSubmind] = Field( - description="Dict of submind `user_id` to `ConnectedSubmind` object") + description="Dict of `submind_id` to `ConnectedSubmind` object" + ) cid_submind_bans: Dict[str, List[str]] = Field( - description="Dict of `cid` to list of banned submind `user_id`s") + description="Dict of `cid` to list of banned `submind_id`s" + ) banned_subminds: List[str] = Field( - description="List of globally banned submind `user_id`s") + description="List of globally banned `submind_id`s" + ) msg_type: Literal["subminds_state"] = Field( - "subminds_state", description="Message type for SIO", deprecated=True) + "subminds_state", description="Message type for SIO", deprecated=True + ) class ChatbotsMqConfiguredPersonasRequest(MQContext): service_name: str = Field( - description="Name of the service to get personas for") + description="Name of the service to get personas for" + ) user_id: Optional[str] = Field( - default=None, description="Optional user_id making with the request.") + default=None, description="Optional user_id making the request." + ) class ChatbotsMqConfiguredPersonasResponse(MQContext): update_time: datetime = Field( - description="Time the personas were last checked") + description="Time the personas were last checked" + ) items: List[LLMPersona] = Field( - description="List of configured personas from Klat") + description="List of configured personas from Klat" + ) context: dict = Field(deprecated=True) - @model_validator(mode='before') + @model_validator(mode="before") @classmethod def validate_context(cls, values): # Deprecated context handling for backwards-compat. - if 'context' not in values and 'message_id' in values: - values['context'] = {"mq": {"message_id": values['message_id']}} + if "context" not in values and "message_id" in values: + values["context"] = {"mq": {"message_id": values["message_id"]}} return values - + def model_dump(self, **kwargs): """ - Override model_dump to include 'persona_name' field for each item based + Override model_dump to include 'persona_name' field for each item based on its 'name' for backwards-compat. with Klat server """ by_alias = {} - if 'by_alias' not in kwargs: + if "by_alias" not in kwargs: # `by_alias` to include `persona_name` in serialized `LLMPersona`s by_alias = super().model_dump(by_alias=True, **kwargs) - + return {**super().model_dump(**kwargs), **by_alias} @classmethod - def from_persona_request(cls, data: dict, - request: ChatbotsMqConfiguredPersonasRequest): - data["items"] = [item for item in data["items"] - if request.service_name in item["supported_llms"]] - return cls(**data, message_id=request.message_id, - routing_key=request.routing_key) + def from_persona_request( + cls, data: dict, request: ChatbotsMqConfiguredPersonasRequest + ): + data["items"] = [ + item + for item in data["items"] + if request.service_name in item["supported_llms"] + ] + return cls( + **data, + message_id=request.message_id, + routing_key=request.routing_key, + ) class ChatbotsMqPromptsDataRequest(MQContext): @@ -371,87 +504,120 @@ class ChatbotsMqPromptsDataResponse(MQContext): context: dict = Field(deprecated=True) - @model_validator(mode='before') + @model_validator(mode="before") @classmethod def validate_context(cls, values): # Deprecated context handling for backwards-compat. - if 'context' not in values and 'message_id' in values: - values['context'] = {"mq": {"message_id": values['message_id']}} + if "context" not in values and "message_id" in values: + values["context"] = {"mq": {"message_id": values["message_id"]}} return values - + @classmethod - def from_prompt_data_request(cls, data: dict, - request: ChatbotsMqPromptsDataRequest): - return cls(**data, message_id=request.message_id, - routing_key=request.routing_key) + def from_prompt_data_request( + cls, data: dict, request: ChatbotsMqPromptsDataRequest + ): + return cls( + **data, + message_id=request.message_id, + routing_key=request.routing_key, + ) class ChatbotsMqSubmindConnection(MQContext): - user_id: str = Field(description="User ID of the submind", alias="nick") + user_id: str = Field( + description="User ID of the submind", + validation_alias="userID", + alias="userID", + ) time: datetime = Field( - default= datetime.now(tz=timezone.utc), - description="Timestamp when the submind last connected") + default=datetime.now(tz=timezone.utc), + description="Timestamp when the submind last connected", + ) cids: Optional[List[str]] = Field( - default=None, description="List of conversation IDs the submind is in") + default=None, description="List of conversation IDs the submind is in" + ) context: Optional[ConnectedSubmind] = Field( default=None, - description="ConnectedSubmind definition of the connecting bot") + description="ConnectedSubmind definition of the connecting bot", + ) - @model_validator(mode='before') + @model_validator(mode="before") @classmethod def validate_context(cls, values): if "context" in values and isinstance(values["context"], dict): - user_id = values.get("user_id") or values.get("nick") or "" - values["context"].setdefault("service_name", - user_id.rsplit('-',1)[0]) + user_id = values.get("user_id") or values.get("userID") or "" + values["context"].setdefault( + "service_name", user_id.rsplit("-", 1)[0] + ) return values class ChatbotsMqSubmindDisconnection(MQContext): - user_id: str = Field(description="User ID of the submind", alias="nick") + user_id: str = Field( + description="User ID of the submind", + validation_alias="userID", + alias="userID", + ) class ChatbotsMqSubmindInvitation(MQContext): cid: str = Field(description="Conversation ID to invite subminds to") requested_participants: List[str] = Field( - description="List of submind User IDs to invite to the conversation") + description="List of submind User IDs to invite to the conversation" + ) class ChatbotsMqUpdateParticipatingSubminds(MQContext): cid: str = Field(description="Conversation ID to update") subminds_to_invite: List[str] = Field( default=[], - description="List of submind User IDs to invite to the conversation") + description="List of submind User IDs to invite to the conversation", + ) subminds_to_kick: List[str] = Field( default=[], - description="List of submind User IDs to evict from the conversation") + description="List of submind User IDs to evict from the conversation", + ) class ChatbotsMqSubmindConversationBan(MQContext): - user_id: str = Field(description="User ID of the submind", alias="nick") + user_id: str = Field( + description="User ID of the submind", + validation_alias="userID", + alias="userID", + ) cid: str = Field(description="Conversation ID to (un)ban submind from") class ChatbotsMqSubmindGlobalBan(MQContext): - user_id: str = Field(description="User ID of the submind", alias="nick") + user_id: str = Field( + description="User ID of the submind", + validation_alias="userID", + alias="userID", + ) class ChatbotsMqSubmindResponseError(MQContext): - message: Optional[str] = Field(default=None, alias="msg", - description="Error message") - - -__all__ = [ChatbotsMqRequest.__name__, ChatbotsMqResponse.__name__, - ChatbotsMqSubmindResponse.__name__, ChatbotsMqSavePrompt.__name__, - ChatbotsMqNewPrompt.__name__, ChatbotsMqSubmindsState.__name__, - ChatbotsMqConfiguredPersonasRequest.__name__, - ChatbotsMqConfiguredPersonasResponse.__name__, - ChatbotsMqPromptsDataRequest.__name__, - ChatbotsMqPromptsDataResponse.__name__, - ChatbotsMqSubmindConnection.__name__, - ChatbotsMqSubmindDisconnection.__name__, - ChatbotsMqSubmindInvitation.__name__, - ChatbotsMqUpdateParticipatingSubminds.__name__, - ChatbotsMqSubmindConversationBan.__name__, - ChatbotsMqSubmindGlobalBan.__name__, - ChatbotsMqSubmindResponseError.__name__,] + message: Optional[str] = Field( + default=None, alias="msg", description="Error message" + ) + + +__all__ = [ + ChatbotsMqRequest.__name__, + ChatbotsMqResponse.__name__, + ChatbotsMqSubmindResponse.__name__, + ChatbotsMqSavePrompt.__name__, + ChatbotsMqNewPrompt.__name__, + ChatbotsMqSubmindsState.__name__, + ChatbotsMqConfiguredPersonasRequest.__name__, + ChatbotsMqConfiguredPersonasResponse.__name__, + ChatbotsMqPromptsDataRequest.__name__, + ChatbotsMqPromptsDataResponse.__name__, + ChatbotsMqSubmindConnection.__name__, + ChatbotsMqSubmindDisconnection.__name__, + ChatbotsMqSubmindInvitation.__name__, + ChatbotsMqUpdateParticipatingSubminds.__name__, + ChatbotsMqSubmindConversationBan.__name__, + ChatbotsMqSubmindGlobalBan.__name__, + ChatbotsMqSubmindResponseError.__name__, +] diff --git a/tests/models/api/test_klat.py b/tests/models/api/test_klat.py index fecff9d9..c279ffba 100644 --- a/tests/models/api/test_klat.py +++ b/tests/models/api/test_klat.py @@ -36,28 +36,28 @@ class TestKlatSocketIO(TestCase): def test_get_stt_request(self): """Test GetSttRequest model creation and validation""" from neon_data_models.models.api.klat.socketio import GetSttRequest - + # Test with required fields test_object = GetSttRequest( cid="test_conversation_id", sid="test_shout_id", - user_uid="test_user_id" + user_uid="test_user_id", ) self.assertIsInstance(test_object, GetSttRequest) self.assertEqual(test_object.cid, "test_conversation_id") self.assertEqual(test_object.sid, "test_shout_id") self.assertEqual(test_object.user_uid, "test_user_id") self.assertEqual(test_object.lang, "en-us") # Default value - + # Test with alias parameter test_object = GetSttRequest( cid="test_conversation_id", message_id="test_message_id", - user_id="test_user_uid" + user_id="test_user_uid", ) self.assertEqual(test_object.sid, "test_message_id") self.assertEqual(test_object.user_uid, "test_user_uid") - + # Test serialization/deserialization dumped = test_object.model_dump() recreated = GetSttRequest(**dumped) @@ -66,12 +66,12 @@ def test_get_stt_request(self): def test_get_stt_response(self): """Test GetSttResponse model creation and validation""" from neon_data_models.models.api.klat.socketio import GetSttResponse - + # Test with required fields test_object = GetSttResponse( transcript="Hello world", sid="test_shout_id", - cid="test_conversation_id" + cid="test_conversation_id", ) self.assertIsInstance(test_object, GetSttResponse) self.assertEqual(test_object.transcript, "Hello world") @@ -79,13 +79,10 @@ def test_get_stt_response(self): self.assertEqual(test_object.cid, "test_conversation_id") self.assertEqual(test_object.lang, "en-us") # Default value self.assertEqual(test_object.context, {}) # Default value - + # Test with context extraction context = {"sid": "context_sid", "cid": "context_cid", "extra": "data"} - test_object = GetSttResponse( - transcript="Hello world", - context=context - ) + test_object = GetSttResponse(transcript="Hello world", context=context) self.assertEqual(test_object.sid, "context_sid") self.assertEqual(test_object.cid, "context_cid") self.assertEqual(test_object.context["extra"], "data") @@ -98,7 +95,7 @@ def test_get_stt_response(self): def test_get_tts_request(self): """Test GetTtsRequest model creation and validation""" from neon_data_models.models.api.klat.socketio import GetTtsRequest - + # Test with required fields test_object = GetTtsRequest( cid="test_conversation_id", @@ -110,7 +107,7 @@ def test_get_tts_request(self): self.assertEqual(test_object.sid, "test_shout_id") self.assertEqual(test_object.user_uid, "test_user_uid") self.assertEqual(test_object.lang, "en-us") # Default value - + # Test with alias parameters test_object = GetTtsRequest( cid="test_conversation_id", @@ -119,7 +116,7 @@ def test_get_tts_request(self): ) self.assertEqual(test_object.sid, "test_message_id") self.assertEqual(test_object.user_uid, "test_user_uid") - + # Test serialization/deserialization dumped = test_object.model_dump() recreated = GetTtsRequest(**dumped) @@ -128,15 +125,15 @@ def test_get_tts_request(self): def test_get_tts_response(self): """Test GetTtsResponse model creation and validation""" from neon_data_models.models.api.klat.socketio import GetTtsResponse - + # Mock base64 audio data mock_audio = base64.b64encode(b"mock audio data").decode() - + # Test with required fields test_object = GetTtsResponse( audio_data=mock_audio, sid="test_shout_id", - cid="test_conversation_id" + cid="test_conversation_id", ) self.assertIsInstance(test_object, GetTtsResponse) self.assertEqual(test_object.audio_data, mock_audio) @@ -145,23 +142,20 @@ def test_get_tts_response(self): self.assertEqual(test_object.lang, "en-us") # Default value self.assertEqual(test_object.gender, "undefined") # Default value self.assertEqual(test_object.context, {}) # Default value - + # Test with context extraction context = {"sid": "context_sid", "cid": "context_cid"} - test_object = GetTtsResponse( - audio_data=mock_audio, - context=context - ) + test_object = GetTtsResponse(audio_data=mock_audio, context=context) self.assertEqual(test_object.sid, "context_sid") self.assertEqual(test_object.cid, "context_cid") - + # Test to_db_query method db_query = test_object.to_db_query() self.assertEqual(db_query["shout_id"], test_object.sid) self.assertEqual(db_query["audio_data"], test_object.audio_data) self.assertEqual(db_query["lang"], test_object.lang) self.assertEqual(db_query["gender"], test_object.gender) - + # Test to_incoming_tts method incoming_tts = test_object.to_incoming_tts() self.assertEqual(incoming_tts["cid"], test_object.cid) @@ -178,14 +172,14 @@ def test_get_tts_response(self): def test_new_prompt_message(self): """Test NewPromptMessage model creation and validation""" from neon_data_models.models.api.klat.socketio import NewPromptMessage - + # Test with required fields test_object = NewPromptMessage( cid="test_conversation_id", user_id="test_user-suffix", user_uid="test_user_uuid", prompt_id="test_prompt_id", - message_text="Hello world" + message_text="Hello world", ) self.assertIsInstance(test_object, NewPromptMessage) self.assertEqual(test_object.cid, "test_conversation_id") @@ -193,9 +187,11 @@ def test_new_prompt_message(self): self.assertEqual(test_object.user_uid, "test_user_uuid") self.assertEqual(test_object.prompt_id, "test_prompt_id") self.assertEqual(test_object.message_text, "Hello world") - self.assertEqual(test_object.prompt_state, CcaiState.IDLE) # Default value + self.assertEqual( + test_object.prompt_state, CcaiState.IDLE + ) # Default value self.assertEqual(test_object.context, {}) # Default value - + # Test with alias parameters test_object = NewPromptMessage( cid="test_conversation_id", @@ -203,13 +199,13 @@ def test_new_prompt_message(self): user_uid="test_user_uuid", promptID="test_prompt_id", messageText="Hello world", - promptState=CcaiState.RESP + promptState=CcaiState.RESP, ) self.assertEqual(test_object.user_id, "test_user-suffix") self.assertEqual(test_object.prompt_id, "test_prompt_id") self.assertEqual(test_object.message_text, "Hello world") self.assertEqual(test_object.prompt_state, CcaiState.RESP) - + # Test model_dump with aliases dumped = test_object.model_dump() self.assertIn("user_id", dumped) @@ -222,11 +218,11 @@ def test_new_prompt_message(self): # Test serialization/deserialization recreated = NewPromptMessage(**dumped) self.assertEqual(test_object, recreated) - + def test_user_message(self): """Test UserMessage model creation and validation""" from neon_data_models.models.api.klat.socketio import UserMessage - + # Test with required fields current_time = datetime.now() test_object = UserMessage( @@ -246,20 +242,20 @@ def test_user_message(self): self.assertEqual(test_object.source, "unknown") # Default value self.assertEqual(test_object.is_bot, "0") # Default value self.assertEqual(test_object.lang, "en") # Default value - + # Test with alias parameters and username derivation test_object = UserMessage( sid="test_shout_id", cid="test_conversation_id", userID="test_user-suffix", messageText="Hello world", - time_created=current_time + time_created=current_time, ) self.assertEqual(test_object.user_id, "test_user-suffix") self.assertEqual(test_object.message_body, "Hello world") # Username should be derived from user_id self.assertEqual(test_object.username, "test_user") - + # Test with validation try: UserMessage( @@ -268,12 +264,12 @@ def test_user_message(self): user_id="same_name", userDisplayName="same_name", # Should raise error when same as user_id message_body="Hello world", - time_created=current_time + time_created=current_time, ) self.fail("Should have raised ValueError") except ValueError as e: self.assertIn("user_id should be a nick + suffix", str(e)) - + # Test isAnnouncement and isAudio aliases test_object = UserMessage( sid="test_shout_id", @@ -282,22 +278,24 @@ def test_user_message(self): time_created=current_time, isAnnouncement=1, isAudio=1, - username="test_user" + username="test_user", ) self.assertTrue(test_object.is_announcement) self.assertTrue(test_object.is_audio) self.assertEqual(test_object.username, "test_user") self.assertIsNone(test_object.user_id) # Derived from username - + # Test to_db_query method db_query = test_object.to_db_query() self.assertEqual(db_query["_id"], test_object.sid) self.assertEqual(db_query["cid"], test_object.cid) self.assertEqual(db_query["message_text"], test_object.message_body) self.assertEqual(db_query["is_audio"], test_object.is_audio) - self.assertEqual(db_query["is_announcement"], test_object.is_announcement) + self.assertEqual( + db_query["is_announcement"], test_object.is_announcement + ) self.assertEqual(db_query["created_on"], int(current_time.timestamp())) - + # Test to_new_prompt_message conversion test_object = UserMessage( sid="test_shout_id", @@ -317,68 +315,125 @@ def test_user_message(self): self.assertEqual(prompt_message.prompt_state, test_object.prompt_state) self.assertEqual(prompt_message.message_text, test_object.message_body) + # Real message + data = { + "routing_key": None, + "message_id": "f29bc127ebf145f9b926424ee48f9f5b", + "sid": "653195bd024c43cdbc1a6c0e8c1407ad", + "cid": "f4ef56bf4b", + "title": "", + "user_id": "logistics_vllm-ae2191f1a93f48809f6a8a29544f3614", + "username": "logistics_vllm", + "message_text": "I am ready for the next prompt", + "replied_message": None, + "bot": "1", + "prompt_id": "", + "is_announcement": False, + "time_created": 1760117318.0, + "source": "klat_observer", + "bot_type": None, + "service_name": None, + "context": {}, + "dom": None, + "omit_reply": True, + "no_save": False, + "to_discussion": False, + "prompt_state": 5, + "messageID": "653195bd024c43cdbc1a6c0e8c1407ad", + "userID": "logistics_vllm-ae2191f1a93f48809f6a8a29544f3614", + "is_bot": "1", + "promptID": "", + "isAnnouncement": "0", + "userDisplayName": "logistics_vllm", + "nick": "logistics_vllm", + "messageText": "I am ready for the next prompt", + "shout": "I am ready for the next prompt", + "repliedMessage": None, + "responded_shout": None, + "timeCreated": 1760117318.0, + "time": 1760117318.0, + "created_on": 1760117318.0, + "promptState": 5, + "conversation_state": 5, + } + message = UserMessage(**data) + self.assertEqual(message.user_id, data["user_id"]) + self.assertEqual(message.username, data["nick"]) + def test_new_ccai_prompt(self): """Test NewCcaiPrompt model creation and validation""" from neon_data_models.models.api.klat.socketio import NewCcaiPrompt - + # Test with required fields test_object = NewCcaiPrompt( prompt_text="What is the meaning of life?", cid="test_conversation_id", - prompt_id="test_prompt_id" + prompt_id="test_prompt_id", ) self.assertIsInstance(test_object, NewCcaiPrompt) - self.assertEqual(test_object.prompt_text, "What is the meaning of life?") + self.assertEqual( + test_object.prompt_text, "What is the meaning of life?" + ) self.assertEqual(test_object.cid, "test_conversation_id") self.assertEqual(test_object.prompt_id, "test_prompt_id") self.assertIsNotNone(test_object.created_on) self.assertEqual(test_object.context, {}) # Default value self.assertIsNone(test_object.winner) # Default value - self.assertEqual(test_object.participating_subminds, []) # Default value + self.assertEqual( + test_object.participating_subminds, [] + ) # Default value self.assertEqual(test_object.proposed_responses, {}) # Default value self.assertEqual(test_object.votes, {}) # Default value - self.assertEqual(test_object.submind_discussion_history, []) # Default value - + self.assertEqual( + test_object.submind_discussion_history, [] + ) # Default value + # Test with None context (should convert to empty dict) test_object = NewCcaiPrompt( prompt_text="What is the meaning of life?", cid="test_conversation_id", prompt_id="test_prompt_id", - context=None + context=None, ) self.assertEqual(test_object.context, {}) - + # Test submind_opinions conversion to submind_discussion_history submind_opinions = {"submind1": "opinion1", "submind2": "opinion2"} test_object = NewCcaiPrompt( prompt_text="What is the meaning of life?", cid="test_conversation_id", prompt_id="test_prompt_id", - submind_opinions=submind_opinions + submind_opinions=submind_opinions, ) self.assertEqual(len(test_object.submind_discussion_history), 1) - self.assertEqual(test_object.submind_discussion_history[0], submind_opinions) - + self.assertEqual( + test_object.submind_discussion_history[0], submind_opinions + ) + # Test to_db_query method db_query = test_object.to_db_query() self.assertEqual(db_query["_id"], test_object.prompt_id) self.assertEqual(db_query["cid"], test_object.cid) self.assertEqual(db_query["is_completed"], "0") - self.assertEqual(db_query["data"]["prompt_text"], test_object.prompt_text) + self.assertEqual( + db_query["data"]["prompt_text"], test_object.prompt_text + ) self.assertEqual(db_query["created_on"], test_object.created_on) self.assertEqual(db_query["context"], test_object.context) def test_ccai_prompt_completed(self): """Test CcaiPromptCompleted model creation and validation""" - from neon_data_models.models.api.klat.socketio import CcaiPromptCompleted - + from neon_data_models.models.api.klat.socketio import ( + CcaiPromptCompleted, + ) + # Prepare a context with required fields current_time = datetime.now() context = { "winner": "The meaning of life is 42", - "prompt": {"prompt_id": "test_prompt_id"} + "prompt": {"prompt_id": "test_prompt_id"}, } - + # Test with required fields test_object = CcaiPromptCompleted( sid="test_shout_id", @@ -387,7 +442,7 @@ def test_ccai_prompt_completed(self): time_created=current_time, prompt_id="test_prompt_id", user_id="proctor-sid", - context=context + context=context, ) self.assertIsInstance(test_object, CcaiPromptCompleted) self.assertEqual(test_object.sid, "test_shout_id") @@ -397,8 +452,10 @@ def test_ccai_prompt_completed(self): self.assertEqual(test_object.prompt_id, "test_prompt_id") self.assertEqual(test_object.winner, "The meaning of life is 42") self.assertEqual(test_object.user_id, "proctor-sid") - self.assertEqual(test_object.username, "proctor") # Parsed from user_id - + self.assertEqual( + test_object.username, "proctor" + ) # Parsed from user_id + # Test with username validation with self.assertRaises(ValidationError): test_object = CcaiPromptCompleted( @@ -408,9 +465,9 @@ def test_ccai_prompt_completed(self): username="same_name", message_body="Hello world", time_created=current_time, - prompt_id="test_prompt_id" + prompt_id="test_prompt_id", ) - + # Test missing prompt_id handling with self.assertRaises(ValidationError): CcaiPromptCompleted( @@ -418,14 +475,14 @@ def test_ccai_prompt_completed(self): cid="test_conversation_id", message_body="Hello world", time_created=current_time, - prompt_id="" # Empty string should raise assertion + prompt_id="", # Empty string should raise assertion ) - + # Test to_db_query method db_query = test_object.to_db_query() self.assertEqual(db_query["prompt_id"], test_object.prompt_id) self.assertEqual(db_query["prompt_context"], test_object.context) - + # Test model_dump with aliases dumped = test_object.model_dump() self.assertIn("prompt_id", dumped) @@ -435,39 +492,38 @@ def test_ccai_prompt_completed(self): def test_get_prompt_data(self): """Test GetPromptData model creation and validation""" from neon_data_models.models.api.klat.socketio import GetPromptData - + # Test with required fields test_object = GetPromptData( nick="test_user", cid="test_conversation_id", - prompt_id="test_prompt_id" + prompt_id="test_prompt_id", ) self.assertIsInstance(test_object, GetPromptData) self.assertEqual(test_object.nick, "test_user") self.assertEqual(test_object.cid, "test_conversation_id") self.assertEqual(test_object.prompt_id, "test_prompt_id") self.assertEqual(test_object.limit, 5) # Default value - + # Test with custom limit test_object = GetPromptData( nick="test_user", cid="test_conversation_id", prompt_id="test_prompt_id", - limit=10 + limit=10, ) self.assertEqual(test_object.limit, 10) - + # Test to_db_query method db_query = test_object.to_db_query() self.assertEqual(db_query["cid"], test_object.cid) self.assertEqual(db_query["limit"], test_object.limit) self.assertEqual(db_query["prompt_ids"], [test_object.prompt_id]) self.assertTrue(db_query["fetch_user_data"]) - + # Test with missing prompt_id test_object = GetPromptData( - nick="test_user", - cid="test_conversation_id" + nick="test_user", cid="test_conversation_id" ) self.assertIsNone(test_object.prompt_id) # to_db_query should raise assertion error when prompt_id is None @@ -477,57 +533,66 @@ def test_get_prompt_data(self): def test_prompt_data(self): """Test PromptData model creation and validation""" from neon_data_models.models.api.klat.socketio import PromptData - + # Test _PromptData nested class test_inner_object = PromptData._PromptData( _id="test_id", - is_completed='1' # String value should be converted to boolean + is_completed="1", # String value should be converted to boolean ) self.assertIsInstance(test_inner_object, PromptData._PromptData) self.assertEqual(test_inner_object.id, "test_id") self.assertTrue(test_inner_object.is_completed) - self.assertEqual(test_inner_object.proposed_responses, {}) # Default value - self.assertEqual(test_inner_object.submind_opinions, {}) # Default value + self.assertEqual( + test_inner_object.proposed_responses, {} + ) # Default value + self.assertEqual( + test_inner_object.submind_opinions, {} + ) # Default value self.assertEqual(test_inner_object.votes, {}) # Default value - self.assertEqual(test_inner_object.participating_subminds, []) # Default value - + self.assertEqual( + test_inner_object.participating_subminds, [] + ) # Default value + # Test serialization of _PromptData serialized = test_inner_object.model_dump() self.assertEqual(serialized["_id"], "test_id") self.assertTrue(serialized["is_completed"]) - + # Test with custom serializer to match MongoDB schema serialized = test_inner_object.alias_serialize() self.assertEqual(serialized["_id"], "test_id") - self.assertEqual(serialized["is_completed"], "1") # Should be string "1" - + self.assertEqual( + serialized["is_completed"], "1" + ) # Should be string "1" + # Test full PromptData with single _PromptData object valid_prompt_data = PromptData( data=test_inner_object, receiver="test_user", - cid="test_conversation_id" + cid="test_conversation_id", ) self.assertIsInstance(valid_prompt_data, PromptData) self.assertEqual(valid_prompt_data.data, test_inner_object) self.assertEqual(valid_prompt_data.receiver, "test_user") self.assertEqual(valid_prompt_data.cid, "test_conversation_id") - self.assertIsNotNone(valid_prompt_data.request_id) # Should have auto-generated UUID - + self.assertIsNotNone( + valid_prompt_data.request_id + ) # Should have auto-generated UUID + # Test full PromptData with list of _PromptData objects test_inner_object2 = PromptData._PromptData( - _id="test_id2", - is_completed='0' + _id="test_id2", is_completed="0" ) valid_prompt_data = PromptData( data=[test_inner_object, test_inner_object2], receiver="test_user", - cid="test_conversation_id" + cid="test_conversation_id", ) self.assertIsInstance(valid_prompt_data.data, list) self.assertEqual(len(valid_prompt_data.data), 2) self.assertEqual(valid_prompt_data.data[0], test_inner_object) self.assertEqual(valid_prompt_data.data[1], test_inner_object2) - + # Test serialization/deserialization dumped = valid_prompt_data.model_dump() recreated = PromptData(**dumped) diff --git a/tests/models/api/test_mq.py b/tests/models/api/test_mq.py index ef5fe514..36f619e8 100644 --- a/tests/models/api/test_mq.py +++ b/tests/models/api/test_mq.py @@ -580,7 +580,7 @@ def test_neon_api_message(self): "requested_skill": "stt", "message_body": "base64encodedstring", "client": "test_client", - "nick": "test_user", + "username": "test_user", "cid": "test_session", "sid": "test_shout_id", "timeCreated": 123456789, @@ -610,7 +610,7 @@ def test_neon_api_message(self): "requested_skill": "recognizer", "messageText": "How are you?", "client": "test_client", - "nick": "test_user", + "username": "test_user", "cid": "test_session", "sid": "test_shout_id", "timeCreated": 123456789, @@ -995,7 +995,7 @@ def test_chatbots_mq_request(self): # Test userID fallback sio_message = { - "userID": "user_id", + "username": "user_id", "cid": "test_conversation", "messageText": "Hello from SIO", "timeCreated": current_time, @@ -1011,7 +1011,7 @@ def test_chatbots_mq_submind_response(self): # Test basic initialization with required fields current_time = datetime.now(tz=timezone.utc) valid_kwargs = { - "userID": "user123", + "userID": "user123-uid_suffix", "messageText": "Hello, this is a response", "message_id": "test_message_id", "conversation_state": 0 @@ -1019,14 +1019,14 @@ def test_chatbots_mq_submind_response(self): response = ChatbotsMqResponse(**valid_kwargs) self.assertIsInstance(response, ChatbotsMqSubmindResponse) - self.assertEqual(response.user_id, "user123") + self.assertEqual(response.username, "user123") self.assertEqual(response.message_text, "Hello, this is a response") self.assertEqual(response.bot, "0") # Default value check self.assertFalse(response.is_announcement) # Default value check # Test with all fields including aliases full_kwargs = { - "userID": "user123", + "userID": "user123-uid_suffix", "userDisplayName": "John Doe", "messageText": "Hello, this is a complete response", "messageID": "msg123", @@ -1044,8 +1044,8 @@ def test_chatbots_mq_submind_response(self): full_response = ChatbotsMqResponse(**full_kwargs) self.assertIsInstance(full_response, ChatbotsMqSubmindResponse) - self.assertEqual(full_response.user_id, "user123") - self.assertEqual(full_response.username, "John Doe") + self.assertEqual(full_response.user_id, "user123-uid_suffix") + self.assertEqual(full_response.username, "user123") self.assertEqual(full_response.message_text, "Hello, this is a complete response") self.assertEqual(full_response.sid, "msg123") self.assertEqual(full_response.replied_message, "original_msg456") @@ -1174,7 +1174,7 @@ def test_chatbots_mq_save_prompt(self): # Test initialization valid_kwargs = { - "userID": "user123", + "userID": "user123-uid_suffix", "messageText": "Prompt completed", "message_id": "test_message_id", "prompt_id": "prompt123", @@ -1185,7 +1185,7 @@ def test_chatbots_mq_save_prompt(self): } save_prompt = ChatbotsMqSavePrompt(**valid_kwargs) self.assertIsInstance(save_prompt, ChatbotsMqSavePrompt) - self.assertEqual(save_prompt.user_id, "user123") + self.assertEqual(save_prompt.username, "user123") self.assertEqual(save_prompt.message_text, "Prompt completed") self.assertEqual(save_prompt.prompt_id, "prompt123") # self.assertEqual(save_prompt.prompt_text, "Test prompt text") @@ -1195,14 +1195,13 @@ def test_chatbots_mq_save_prompt(self): # Test model_dump inheritance from ChatbotsMqResponse serialized = save_prompt.model_dump() self.assertIn("userID", serialized) - self.assertIn("messageText", serialized) self.assertIn("prompt_id", serialized) self.assertIn("context", serialized) # Test missing required fields with self.assertRaises(ValidationError): ChatbotsMqSavePrompt( - userID="user123", + userID="user123-uid_suffix", messageText="Prompt completed", message_id="test_message_id", prompt_id="prompt123", @@ -1227,10 +1226,9 @@ def test_chatbots_mq_new_prompt(self): new_prompt = ChatbotsMqNewPrompt(**valid_kwargs) self.assertIsInstance(new_prompt, ChatbotsMqNewPrompt) self.assertEqual(new_prompt.user_id, "user123") - self.assertEqual(new_prompt.message_text, valid_kwargs["messageText"]) self.assertEqual(new_prompt.prompt_id, "prompt123") self.assertEqual(new_prompt.prompt_text, "Test prompt text") - self.assertIsNone(new_prompt.context) + self.assertEqual(new_prompt.context, {}) # Test with conversation_context alias context_kwargs = { @@ -1256,12 +1254,10 @@ def test_chatbots_mq_new_prompt(self): "conversation_state": 0 } partial_prompt = ChatbotsMqNewPrompt(**partial_kwargs) - self.assertEqual(partial_prompt.message_text, '') # Test model_dump inheritance from ChatbotsMqResponse - serialized = new_prompt.model_dump() + serialized = partial_prompt.model_dump() self.assertIn("user_id", serialized) - self.assertIn("messageText", serialized) self.assertIn("prompt_id", serialized) # Test missing required fields @@ -1307,7 +1303,7 @@ def test_old_class_names_compatibility(self): self.assertIsInstance(req, ChatbotsMqRequest) resp = ChatbotsMqResponse( - userID="test_user", + userID="test_user-uid_suffix", messageText="Test response", message_id="test_message_id", conversation_state=0, @@ -1610,141 +1606,16 @@ def test_chatbots_mq_prompts_data_response(self): with self.assertRaises(ValidationError): ChatbotsMqPromptsDataResponse(records=["prompt1", "prompt2"]) - def test_chatbots_mq_submind_response(self): - from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindResponse - from datetime import datetime, timezone - - # Test basic initialization with required fields - current_time = datetime.now(tz=timezone.utc) - valid_kwargs = { - "userID": "test_user", - "messageText": "Test submind response", - "message_id": "test_message_id", - "prompt_state": 1, - } - - response = ChatbotsMqSubmindResponse(**valid_kwargs) - self.assertIsInstance(response, ChatbotsMqSubmindResponse) - self.assertEqual(response.user_id, "test_user") - self.assertEqual(response.message_text, "Test submind response") - self.assertEqual(response.bot, "0") # Default value check - - # Test with all fields including aliased fields - full_kwargs = { - "userID": "submind1", - "userDisplayName": "Test Submind", - "messageText": "Full test response", - "messageID": "test_shout_id", - "repliedMessage": "original_message_id", - "bot": "1", - "promptID": "prompt_id_123", - "conversation_state": 2, - "is_announcement": True, - "timeCreated": current_time, - "source": "test_source", - "client": "test_client", - "cid": "conversation123", - "message_id": "test_message_id" - } - - full_response = ChatbotsMqSubmindResponse(**full_kwargs) - self.assertIsInstance(full_response, ChatbotsMqSubmindResponse) - self.assertEqual(full_response.user_id, "submind1") - self.assertEqual(full_response.username, "Test Submind") - self.assertEqual(full_response.message_text, "Full test response") - self.assertEqual(full_response.sid, "test_shout_id") - self.assertEqual(full_response.replied_message, "original_message_id") - self.assertEqual(full_response.bot, "1") - self.assertEqual(full_response.prompt_id, "prompt_id_123") - self.assertEqual(full_response.prompt_state, 2) - self.assertTrue(full_response.is_announcement) - self.assertEqual(full_response.time_created, current_time) - self.assertEqual(full_response.source, "test_source") - - # Test alternate field names (validate_inputs validator) - alternate_kwargs = { - "nick": "submind2", - "shout": "Message using alternate field names", - "responded_shout": "parent_message_id", - "time": current_time.timestamp(), - "message_id": "test_message_id", - "conversation_state": 2, - } - - alternate_response = ChatbotsMqSubmindResponse(**alternate_kwargs) - self.assertEqual(alternate_response.user_id, "submind2") - self.assertEqual(alternate_response.message_text, "Message using alternate field names") - self.assertEqual(alternate_response.replied_message, "parent_message_id") - - # Test model_dump serialization - serialized = full_response.model_dump(by_alias=True) - self.assertEqual(serialized["userID"], "submind1") - self.assertEqual(serialized["messageText"], "Full test response") - self.assertTrue(serialized["isAnnouncement"]) - self.assertEqual(serialized["nick"], "submind1") # Check backwards compatibility field - self.assertEqual(serialized["shout"], "Full test response") # Check backwards compatibility field - - - # Test username from user_id with a single hyphen - hyphen_user_id = ChatbotsMqSubmindResponse( - userID="service-123456", - messageText="Test message", - message_id="test_id" - ) - self.assertEqual(hyphen_user_id.username, "service") - - # Test username from user_id with multiple hyphens - multiple_hyphens = ChatbotsMqSubmindResponse( - userID="service-name-with-hyphens-123456", - messageText="Test message", - message_id="test_id" - ) - self.assertEqual(multiple_hyphens.username, "service-name-with-hyphens") - - # Test username from user_id without hyphen - no_hyphen = ChatbotsMqSubmindResponse( - userID="servicename", - messageText="Test message", - message_id="test_id" - ) - self.assertEqual(no_hyphen.username, "servicename") - - # Test existing username is preserved when both fields are provided - preserved_username = ChatbotsMqSubmindResponse( - userID="service-123456", - username="custom_name", - messageText="Test message", - message_id="test_id" - ) - self.assertEqual(preserved_username.username, "custom_name") - - # Test when user_id is None - with self.assertRaises(ValidationError) as e: - none_user_id = ChatbotsMqSubmindResponse( - userID=None, - messageText="Test message", - message_id="test_id" - ) - self.assertEqual(str(e), "user_id cannot be None") - - # Test missing required fields - with self.assertRaises(ValidationError): - ChatbotsMqSubmindResponse(messageText="Missing user_id", message_id="test_id") - - with self.assertRaises(ValidationError): - ChatbotsMqSubmindResponse(userID="test_user", message_id="test_id") - def test_chatbots_mq_response_type_adapter(self): from neon_data_models.models.api.mq.chatbots import ChatbotsMqResponse from neon_data_models.models.api.mq.chatbots import ChatbotsMqSubmindResponse, ChatbotsMqSavePrompt, ChatbotsMqNewPrompt - from neon_data_models.enum import CcaiControl from datetime import datetime, timezone current_time = datetime.now(tz=timezone.utc) # Test regular submind response regular_message = { - "userID": "submind1", + "userID": "submind1-uid_suffix", "messageText": "Regular response", "conversation_state": 2, "message_id": "test_message_id" @@ -1752,7 +1623,8 @@ def test_chatbots_mq_response_type_adapter(self): result = ChatbotsMqResponse(**regular_message) self.assertIsInstance(result, ChatbotsMqSubmindResponse) - self.assertEqual(result.user_id, "submind1") + self.assertEqual(result.user_id, "submind1-uid_suffix") + self.assertEqual(result.username, "submind1") self.assertEqual(result.message_text, "Regular response") # Test new prompt message @@ -1783,7 +1655,7 @@ def test_chatbots_mq_response_type_adapter(self): } alt_new_prompt_message = { - "nick": "proctor-ac18f03d0937490080c798d3b242ecd0", + "user_id": "proctor-ac18f03d0937490080c798d3b242ecd0", "bot_type": "proctor", "service_name": "proctor", "cid": "35d5dff220", @@ -1808,7 +1680,7 @@ def test_chatbots_mq_response_type_adapter(self): alt_response = ChatbotsMqResponse(**alt_new_prompt_message) self.assertIsInstance(alt_response, ChatbotsMqNewPrompt) - self.assertEqual(alt_response.user_id, alt_new_prompt_message['nick']) + self.assertEqual(alt_response.user_id, alt_new_prompt_message['user_id']) # Test save prompt message @@ -1839,7 +1711,7 @@ def test_chatbots_mq_response_type_adapter(self): "no_save": False } alt_save_prompt_message = { - "nick": "proctor-ac18f03d0937490080c798d3b242ecd0", + "user_id": "proctor-ac18f03d0937490080c798d3b242ecd0", "bot_type": "proctor", "service_name": "proctor", "cid": "35d5dff220", @@ -1873,7 +1745,7 @@ def test_chatbots_mq_response_type_adapter(self): "bound_service": "", "bot": "0", "messageText": "Why is testing important?", - "nick": "ca45d1ea45134523af7f" + "user_uid": "ca45d1ea45134523af7f" }, "is_active": True, "prompt_text": "Why is testing important?", @@ -1929,7 +1801,7 @@ def test_chatbots_mq_response_type_adapter(self): # Test with message_text vs messageText alternate_syntax = { - "userID": "submind1", + "userID": "submind1-uid_suffix", "message_text": "Using message_text instead of messageText", "conversation_state": 2, "message_id": "test_message_id" @@ -1946,7 +1818,7 @@ def test_chatbots_mq_submind_connection(self): # Test basic initialization current_time = datetime.now(tz=timezone.utc) valid_kwargs = { - "nick": "submind1", + "userID": "submind1", "time": current_time, "message_id": "test_message_id" } @@ -1994,7 +1866,7 @@ def test_chatbots_mq_submind_disconnection(self): # Test basic initialization valid_kwargs = { - "nick": "submind1", + "userID": "submind1", "message_id": "test_message_id" } @@ -2077,7 +1949,7 @@ def test_chatbots_mq_submind_conversation_ban(self): # Test basic initialization valid_kwargs = { - "nick": "submind1", + "user_id": "submind1", "cid": "conversation123", "message_id": "test_message_id" } @@ -2110,7 +1982,7 @@ def test_chatbots_mq_submind_global_ban(self): # Test basic initialization valid_kwargs = { - "nick": "submind1", + "user_id": "submind1", "message_id": "test_message_id" } @@ -2172,7 +2044,7 @@ def test_chatbots_mq_response_edge_cases(self): # Test with empty/minimal valid message minimal_message = { - "userID": "test_user", + "userID": "test_user-uid_suffix", "messageText": "", "message_id": "test_message_id" } @@ -2183,20 +2055,20 @@ def test_chatbots_mq_response_edge_cases(self): # Test with mixed aliased and non-aliased fields mixed_fields = { - "userID": "test_user", + "userID": "test_user-uid_suffix", "shout": "Using aliased shout field", "message_id": "test_message_id", - "nick": "nick_value" # This should be used for user_id too + # "nick": "nick_value" # This should not be used } result = ChatbotsMqResponse(**mixed_fields) self.assertIsInstance(result, ChatbotsMqSubmindResponse) self.assertEqual(result.message_text, "Using aliased shout field") - self.assertEqual(result.user_id, "test_user") # userID takes precedence over nick + self.assertEqual(result.username, "test_user") # userID takes precedence over nick # Test with malformed timestamps that should be converted timestamp_message = { - "userID": "test_user", + "userID": "test_user-uid_suffix", "messageText": "Timestamp test", "created_on": "1611234567", # String timestamp "time": 1611234567, # Integer timestamp @@ -2211,7 +2083,7 @@ def test_chatbots_mq_response_edge_cases(self): # Test with null/None fields that should be handled null_fields = { - "userID": "test_user", + "userID": "test_user-uid_suffix", "messageText": "Test with nulls", "message_id": "test_message_id", "sid": None, # Should be handled by the model validator From a0dc6fe68b2e42dfaa978fed70a2705b90af90bc Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Tue, 4 Nov 2025 02:09:06 +0000 Subject: [PATCH 30/36] Increment Version to 0.0.2a15 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 66fbf914..395a471e 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a14" +__version__ = "0.0.2a15" From 548f8b3dc600682bb7272963fc848d327ac8e661 Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Wed, 28 Jan 2026 09:15:04 -0800 Subject: [PATCH 31/36] Support more message roles in LLM requests (#31) * Support "information" and "instruction" roles * Update `LLMResponse` model to support new message types * Refactor LLM roles to an enum * Add "tool" role --- neon_data_models/models/api/llm.py | 5 +++-- neon_data_models/types.py | 1 + 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/neon_data_models/models/api/llm.py b/neon_data_models/models/api/llm.py index e8e460e0..94683a07 100644 --- a/neon_data_models/models/api/llm.py +++ b/neon_data_models/models/api/llm.py @@ -27,6 +27,7 @@ from pydantic import Field, model_validator, computed_field from neon_data_models.models.base import BaseModel +from neon_data_models.types import LlmMessageRole _DEFAULT_MQ_TO_ROLE = {"user": "user", "llm": "assistant"} @@ -80,7 +81,7 @@ def validate_request(self): class LLMRequest(BaseModel): query: str = Field(description="Incoming user prompt") # TODO: History may support more options in the future - history: List[Tuple[Literal["user", "llm"], str]] = Field( + history: List[Tuple[LlmMessageRole, str]] = Field( description="Formatted chat history (excluding system prompt). Note " "that the roles used here will differ from those used in " "OpenAI-compatible requests.") @@ -201,7 +202,7 @@ def to_completion_kwargs(self, mq2role: dict = None) -> dict: class LLMResponse(BaseModel): response: str = Field(description="LLM Response to the input query") - history: List[Tuple[Literal["user", "llm"], str]] = Field( + history: List[Tuple[LlmMessageRole, str]] = Field( description="List of (role, content) tuples in chronological order " "(`response` is in the last list element)") finish_reason: Literal["length", "stop"] = Field( diff --git a/neon_data_models/types.py b/neon_data_models/types.py index 62b9a7dd..282e5bdc 100644 --- a/neon_data_models/types.py +++ b/neon_data_models/types.py @@ -29,4 +29,5 @@ Gender = Literal["male", "female"] BotType = Literal["submind", "facilitator", "proctor", "observer"] +LlmMessageRole = Literal["user", "llm", "information", "instruction", "assistant", "tool"] # TODO: consider deprecating special facilitator types in `BotType` From dd3e583131e23ed9838ee7b5195ff2bd011457b2 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Wed, 28 Jan 2026 17:15:22 +0000 Subject: [PATCH 32/36] Increment Version to 0.0.2a16 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 395a471e..59d1e4b1 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a15" +__version__ = "0.0.2a16" From b5b34594854f28dbf840cf85bf0321fcf49ab560 Mon Sep 17 00:00:00 2001 From: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Thu, 29 Jan 2026 17:48:41 -0800 Subject: [PATCH 33/36] Minor updates to prep release (#32) * Update license notices to 2026 * Add missing top-level imports for consistency * Update TODO comments in `types` module --- LICENSE.md | 4 ++-- neon_data_models/__init__.py | 2 +- neon_data_models/enum.py | 2 +- neon_data_models/models/__init__.py | 2 +- neon_data_models/models/api/__init__.py | 6 ++++-- neon_data_models/models/api/chatbots.py | 2 +- neon_data_models/models/api/http/__init__.py | 1 + neon_data_models/models/api/http/neon.py | 2 +- neon_data_models/models/api/jwt.py | 2 +- neon_data_models/models/api/klat/socketio.py | 2 +- neon_data_models/models/api/llm.py | 2 +- neon_data_models/models/api/messagebus/__init__.py | 2 +- neon_data_models/models/api/mq/__init__.py | 2 +- neon_data_models/models/api/mq/chatbots.py | 2 +- neon_data_models/models/api/mq/llm.py | 2 +- neon_data_models/models/api/mq/neon.py | 2 +- neon_data_models/models/api/mq/users.py | 2 +- neon_data_models/models/api/node_v1/__init__.py | 2 +- neon_data_models/models/base/__init__.py | 2 +- neon_data_models/models/base/contexts.py | 2 +- neon_data_models/models/base/messagebus.py | 2 +- neon_data_models/models/client/__init__.py | 2 +- neon_data_models/models/client/node.py | 2 +- neon_data_models/models/user/__init__.py | 2 +- neon_data_models/models/user/database.py | 2 +- neon_data_models/models/user/neon_profile.py | 2 +- neon_data_models/types.py | 5 +++-- neon_data_models/util.py | 2 +- neon_data_models/version.py | 2 +- setup.py | 2 +- tests/models/api/test_chatbots.py | 2 +- tests/models/api/test_http.py | 4 ++-- tests/models/api/test_llm.py | 2 +- tests/models/api/test_messagebus.py | 2 +- tests/models/api/test_mq.py | 2 +- tests/models/api/test_node.py | 2 +- tests/models/test_base.py | 2 +- tests/models/test_client.py | 2 +- tests/models/test_user.py | 2 +- tests/test_enum.py | 2 +- tests/test_imports.py | 2 +- 41 files changed, 48 insertions(+), 44 deletions(-) diff --git a/LICENSE.md b/LICENSE.md index 1072bd39..2854c36a 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 Redistribution and use in source and binary forms, with or without modification, are permitted provided that the @@ -18,4 +18,4 @@ DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF -THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. \ No newline at end of file +THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/neon_data_models/__init__.py b/neon_data_models/__init__.py index f6f252ae..40ae3e3b 100644 --- a/neon_data_models/__init__.py +++ b/neon_data_models/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/enum.py b/neon_data_models/enum.py index eface825..a709277d 100644 --- a/neon_data_models/enum.py +++ b/neon_data_models/enum.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/__init__.py b/neon_data_models/models/__init__.py index e6cd0063..ee4c8781 100644 --- a/neon_data_models/models/__init__.py +++ b/neon_data_models/models/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/__init__.py b/neon_data_models/models/api/__init__.py index bbc589a1..efc54056 100644 --- a/neon_data_models/models/api/__init__.py +++ b/neon_data_models/models/api/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -29,4 +29,6 @@ from neon_data_models.models.api.llm import * from neon_data_models.models.api.mq import * from neon_data_models.models.api.messagebus import * -from neon_data_models.models.api.klat import * \ No newline at end of file +from neon_data_models.models.api.klat import * +from neon_data_models.models.api.chatbots import * +from neon_data_models.models.api.http import * diff --git a/neon_data_models/models/api/chatbots.py b/neon_data_models/models/api/chatbots.py index 4f31bbb7..a0d402f9 100644 --- a/neon_data_models/models/api/chatbots.py +++ b/neon_data_models/models/api/chatbots.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/http/__init__.py b/neon_data_models/models/api/http/__init__.py index 19a7ca21..ea8573be 100644 --- a/neon_data_models/models/api/http/__init__.py +++ b/neon_data_models/models/api/http/__init__.py @@ -25,6 +25,7 @@ # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. from neon_data_models.models.api.http.brainforge import * +from neon_data_models.models.api.http.neon import * """ This module contains models for interacting via HANA (HTTP). diff --git a/neon_data_models/models/api/http/neon.py b/neon_data_models/models/api/http/neon.py index cdf22c83..d747c605 100644 --- a/neon_data_models/models/api/http/neon.py +++ b/neon_data_models/models/api/http/neon.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/jwt.py b/neon_data_models/models/api/jwt.py index f6ef9000..f464d5fd 100644 --- a/neon_data_models/models/api/jwt.py +++ b/neon_data_models/models/api/jwt.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/klat/socketio.py b/neon_data_models/models/api/klat/socketio.py index 52363de4..36185f8e 100644 --- a/neon_data_models/models/api/klat/socketio.py +++ b/neon_data_models/models/api/klat/socketio.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/llm.py b/neon_data_models/models/api/llm.py index 94683a07..803d7ce7 100644 --- a/neon_data_models/models/api/llm.py +++ b/neon_data_models/models/api/llm.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/messagebus/__init__.py b/neon_data_models/models/api/messagebus/__init__.py index 2f25326a..14b4e5a8 100644 --- a/neon_data_models/models/api/messagebus/__init__.py +++ b/neon_data_models/models/api/messagebus/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/mq/__init__.py b/neon_data_models/models/api/mq/__init__.py index 219bf706..f7a6b400 100644 --- a/neon_data_models/models/api/mq/__init__.py +++ b/neon_data_models/models/api/mq/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/mq/chatbots.py b/neon_data_models/models/api/mq/chatbots.py index 52a0713f..1f07819c 100644 --- a/neon_data_models/models/api/mq/chatbots.py +++ b/neon_data_models/models/api/mq/chatbots.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/mq/llm.py b/neon_data_models/models/api/mq/llm.py index a40fd599..6cc502bc 100644 --- a/neon_data_models/models/api/mq/llm.py +++ b/neon_data_models/models/api/mq/llm.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/mq/neon.py b/neon_data_models/models/api/mq/neon.py index ddd9cc53..c17b585e 100644 --- a/neon_data_models/models/api/mq/neon.py +++ b/neon_data_models/models/api/mq/neon.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/mq/users.py b/neon_data_models/models/api/mq/users.py index a5b94e19..1e46ba6f 100644 --- a/neon_data_models/models/api/mq/users.py +++ b/neon_data_models/models/api/mq/users.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/api/node_v1/__init__.py b/neon_data_models/models/api/node_v1/__init__.py index 71e9bcd8..b7bdce47 100644 --- a/neon_data_models/models/api/node_v1/__init__.py +++ b/neon_data_models/models/api/node_v1/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/base/__init__.py b/neon_data_models/models/base/__init__.py index b31cba13..d2dbe138 100644 --- a/neon_data_models/models/base/__init__.py +++ b/neon_data_models/models/base/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/base/contexts.py b/neon_data_models/models/base/contexts.py index c48e683a..e66efe4b 100644 --- a/neon_data_models/models/base/contexts.py +++ b/neon_data_models/models/base/contexts.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/base/messagebus.py b/neon_data_models/models/base/messagebus.py index a26a80d4..12cbe11c 100644 --- a/neon_data_models/models/base/messagebus.py +++ b/neon_data_models/models/base/messagebus.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/client/__init__.py b/neon_data_models/models/client/__init__.py index e1d91a69..c19b2cf5 100644 --- a/neon_data_models/models/client/__init__.py +++ b/neon_data_models/models/client/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/client/node.py b/neon_data_models/models/client/node.py index 79880aed..bcce5223 100644 --- a/neon_data_models/models/client/node.py +++ b/neon_data_models/models/client/node.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/user/__init__.py b/neon_data_models/models/user/__init__.py index 8db545b3..da401a59 100644 --- a/neon_data_models/models/user/__init__.py +++ b/neon_data_models/models/user/__init__.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/user/database.py b/neon_data_models/models/user/database.py index 6c92f0d5..d1939e89 100644 --- a/neon_data_models/models/user/database.py +++ b/neon_data_models/models/user/database.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/models/user/neon_profile.py b/neon_data_models/models/user/neon_profile.py index bbb39961..430a5d77 100644 --- a/neon_data_models/models/user/neon_profile.py +++ b/neon_data_models/models/user/neon_profile.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/types.py b/neon_data_models/types.py index 282e5bdc..2831fb01 100644 --- a/neon_data_models/types.py +++ b/neon_data_models/types.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -29,5 +29,6 @@ Gender = Literal["male", "female"] BotType = Literal["submind", "facilitator", "proctor", "observer"] -LlmMessageRole = Literal["user", "llm", "information", "instruction", "assistant", "tool"] # TODO: consider deprecating special facilitator types in `BotType` +LlmMessageRole = Literal["user", "llm", "information", "instruction", "assistant", "tool"] +# TODO: consider deprecating "llm" which always maps to "assistant" in actual requests diff --git a/neon_data_models/util.py b/neon_data_models/util.py index fdd7aef8..6832db59 100644 --- a/neon_data_models/util.py +++ b/neon_data_models/util.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 59d1e4b1..270df5ee 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Framework # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # Contributors: Daniel McKnight, Guy Daniels, Elon Gasper, Richard Leeds, # Regina Bloomstine, Casimiro Ferreira, Andrii Pernatii, Kirill Hrymailo # BSD-3 License diff --git a/setup.py b/setup.py index f402e395..dc64590c 100644 --- a/setup.py +++ b/setup.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Framework # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # Contributors: Daniel McKnight, Guy Daniels, Elon Gasper, Richard Leeds, # Regina Bloomstine, Casimiro Ferreira, Andrii Pernatii, Kirill Hrymailo # BSD-3 License diff --git a/tests/models/api/test_chatbots.py b/tests/models/api/test_chatbots.py index 089459d9..950e5302 100644 --- a/tests/models/api/test_chatbots.py +++ b/tests/models/api/test_chatbots.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/models/api/test_http.py b/tests/models/api/test_http.py index fc4424cc..c1c73a3d 100644 --- a/tests/models/api/test_http.py +++ b/tests/models/api/test_http.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: @@ -427,4 +427,4 @@ def test_neon_http_skill_api_response(self): response_data = valid_response_complex.model_dump() reconstructed = NeonHttpSkillApiResponse(**response_data) self.assertEqual(reconstructed.result, complex_result) - self.assertIsNone(reconstructed.error) \ No newline at end of file + self.assertIsNone(reconstructed.error) diff --git a/tests/models/api/test_llm.py b/tests/models/api/test_llm.py index 153be3cc..8e643143 100644 --- a/tests/models/api/test_llm.py +++ b/tests/models/api/test_llm.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/models/api/test_messagebus.py b/tests/models/api/test_messagebus.py index d52bbef3..e49dd56b 100644 --- a/tests/models/api/test_messagebus.py +++ b/tests/models/api/test_messagebus.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/models/api/test_mq.py b/tests/models/api/test_mq.py index 36f619e8..c2164a78 100644 --- a/tests/models/api/test_mq.py +++ b/tests/models/api/test_mq.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/models/api/test_node.py b/tests/models/api/test_node.py index 255be521..9f297eea 100644 --- a/tests/models/api/test_node.py +++ b/tests/models/api/test_node.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/models/test_base.py b/tests/models/test_base.py index 9f8e8fae..29358838 100644 --- a/tests/models/test_base.py +++ b/tests/models/test_base.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/models/test_client.py b/tests/models/test_client.py index ca7fc5c6..66a75193 100644 --- a/tests/models/test_client.py +++ b/tests/models/test_client.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/models/test_user.py b/tests/models/test_user.py index d03b46ab..7e8161a6 100644 --- a/tests/models/test_user.py +++ b/tests/models/test_user.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/test_enum.py b/tests/test_enum.py index a3f64329..9ffc1c7d 100644 --- a/tests/test_enum.py +++ b/tests/test_enum.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: diff --git a/tests/test_imports.py b/tests/test_imports.py index 4fa67cdc..8c7e1258 100644 --- a/tests/test_imports.py +++ b/tests/test_imports.py @@ -1,6 +1,6 @@ # NEON AI (TM) SOFTWARE, Software Development Kit & Application Development System # All trademark and other rights reserved by their respective owners -# Copyright 2008-2025 Neongecko.com Inc. +# Copyright 2008-2026 Neongecko.com Inc. # BSD-3 # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: From 892b5b17ce648f6a9e21c34ef1d94606561df110 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Fri, 30 Jan 2026 01:48:53 +0000 Subject: [PATCH 34/36] Increment Version to 0.0.2a17 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index 270df5ee..f4fdf5ed 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a16" +__version__ = "0.0.2a17" From 12caa9707ed25e797b80b6a1e8ec7b9525e3393b Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Fri, 30 Jan 2026 02:02:22 +0000 Subject: [PATCH 35/36] Increment Version to 0.0.2 --- neon_data_models/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neon_data_models/version.py b/neon_data_models/version.py index f4fdf5ed..96a1b744 100644 --- a/neon_data_models/version.py +++ b/neon_data_models/version.py @@ -26,4 +26,4 @@ # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -__version__ = "0.0.2a17" +__version__ = "0.0.2" From 2091da00ff5b23d08533f17b12a724d755249624 Mon Sep 17 00:00:00 2001 From: NeonDaniel Date: Fri, 30 Jan 2026 02:02:41 +0000 Subject: [PATCH 36/36] Update Changelog --- CHANGELOG.md | 120 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 86 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e1a920a9..ee4ed312 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,88 +1,140 @@ # Changelog -## [0.0.0a11](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a11) (2025-02-07) +## [0.0.2a17](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a17) (2026-01-30) -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.0a10...0.0.0a11) +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a16...0.0.2a17) -**Implemented enhancements:** +**Merged pull requests:** + +- Minor updates to prep release [\#32](https://github.com/NeonGeckoCom/neon-data-models/pull/32) ([NeonDaniel](https://github.com/NeonDaniel)) + +## [0.0.2a16](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a16) (2026-01-28) + +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a15...0.0.2a16) + +**Merged pull requests:** + +- Support more message roles in LLM requests [\#31](https://github.com/NeonGeckoCom/neon-data-models/pull/31) ([NeonDaniel](https://github.com/NeonDaniel)) + +## [0.0.2a15](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a15) (2025-11-04) + +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a14...0.0.2a15) -- \[FEAT\] Skip system prompt if `persona=None` [\#9](https://github.com/NeonGeckoCom/neon-data-models/issues/9) +**Merged pull requests:** + +- Refactor Chatbots messaging to match Klat [\#30](https://github.com/NeonGeckoCom/neon-data-models/pull/30) ([NeonDaniel](https://github.com/NeonDaniel)) + +## [0.0.2a14](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a14) (2025-10-07) + +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a13...0.0.2a14) **Merged pull requests:** -- Add support for "vanilla" persona with unit tests [\#15](https://github.com/NeonGeckoCom/neon-data-models/pull/15) ([NeonDaniel](https://github.com/NeonDaniel)) +- Refactor UserMessage SIO model [\#29](https://github.com/NeonGeckoCom/neon-data-models/pull/29) ([NeonDaniel](https://github.com/NeonDaniel)) -## [0.0.0a10](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a10) (2025-02-06) +## [0.0.2a13](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a13) (2025-10-07) -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.0a9...0.0.0a10) +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a12...0.0.2a13) **Merged pull requests:** -- Update `llm` submodule and model docstrings [\#13](https://github.com/NeonGeckoCom/neon-data-models/pull/13) ([NeonDaniel](https://github.com/NeonDaniel)) +- Implement Models for Klat SocketIO events [\#11](https://github.com/NeonGeckoCom/neon-data-models/pull/11) ([NeonDaniel](https://github.com/NeonDaniel)) -## [0.0.0a9](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a9) (2025-02-06) +## [0.0.2a12](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a12) (2025-09-25) -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.0a8...0.0.0a9) +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a11...0.0.2a12) -## [0.0.0a8](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a8) (2025-02-02) +**Merged pull requests:** -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.0a7...0.0.0a8) +- Update ChatbotsMqRequest for backwards-compat [\#28](https://github.com/NeonGeckoCom/neon-data-models/pull/28) ([NeonDaniel](https://github.com/NeonDaniel)) -## [0.0.0a7](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a7) (2025-01-26) +## [0.0.2a11](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a11) (2025-08-15) -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.0a6...0.0.0a7) +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a10...0.0.2a11) -## [0.0.0a6](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a6) (2025-01-06) +**Merged pull requests:** + +- Implement models for the Skill API [\#27](https://github.com/NeonGeckoCom/neon-data-models/pull/27) ([NeonDaniel](https://github.com/NeonDaniel)) -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.0a5...0.0.0a6) +## [0.0.2a10](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a10) (2025-08-07) + +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a9...0.0.2a10) **Merged pull requests:** -- Resolve issue with serializing default permissions [\#10](https://github.com/NeonGeckoCom/neon-data-models/pull/10) ([NeonDaniel](https://github.com/NeonDaniel)) +- Implement OpenAI Completion request/response models with unit tests [\#26](https://github.com/NeonGeckoCom/neon-data-models/pull/26) ([NeonDaniel](https://github.com/NeonDaniel)) -## [0.0.0a5](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a5) (2024-12-18) +## [0.0.2a9](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a9) (2025-06-13) -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.0a4...0.0.0a5) +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a8...0.0.2a9) **Merged pull requests:** -- Define data models for LLM interactions [\#4](https://github.com/NeonGeckoCom/neon-data-models/pull/4) ([NeonDaniel](https://github.com/NeonDaniel)) +- Specify `discussion_rounds` in new prompt model [\#23](https://github.com/NeonGeckoCom/neon-data-models/pull/23) ([NeonDaniel](https://github.com/NeonDaniel)) -## [0.0.0a4](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a4) (2024-11-21) +## [0.0.2a8](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a8) (2025-06-13) -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.0a3...0.0.0a4) +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a7...0.0.2a8) **Merged pull requests:** -- Update UserDbRequest models [\#3](https://github.com/NeonGeckoCom/neon-data-models/pull/3) ([NeonDaniel](https://github.com/NeonDaniel)) +- Update `chatbots` MQ models to improve validation and backwards-compat [\#21](https://github.com/NeonGeckoCom/neon-data-models/pull/21) ([NeonDaniel](https://github.com/NeonDaniel)) + +## [0.0.2a7](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a7) (2025-05-31) -## [0.0.0a3](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a3) (2024-11-19) +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a6...0.0.2a7) + +**Merged pull requests:** + +- Add validator to make `LLMGetInferenceHttpRequest` more flexible [\#25](https://github.com/NeonGeckoCom/neon-data-models/pull/25) ([NeonDaniel](https://github.com/NeonDaniel)) + +## [0.0.2a6](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a6) (2025-05-27) + +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a5...0.0.2a6) + +**Merged pull requests:** + +- Update license tests to use default packages-exclude list [\#24](https://github.com/NeonGeckoCom/neon-data-models/pull/24) ([NeonDaniel](https://github.com/NeonDaniel)) + +## [0.0.2a5](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a5) (2025-04-19) + +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a4...0.0.2a5) + +**Merged pull requests:** + +- Fix type annotation for `user_profiles` context to match expected format [\#22](https://github.com/NeonGeckoCom/neon-data-models/pull/22) ([NeonDaniel](https://github.com/NeonDaniel)) + +## [0.0.2a4](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a4) (2025-04-10) + +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a3...0.0.2a4) + +**Merged pull requests:** -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.0a2...0.0.0a3) +- Implement models for BrainForge service [\#8](https://github.com/NeonGeckoCom/neon-data-models/pull/8) ([NeonDaniel](https://github.com/NeonDaniel)) -**Implemented enhancements:** +## [0.0.2a3](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a3) (2025-04-08) -- \[FEAT\] Define range for user permissions [\#5](https://github.com/NeonGeckoCom/neon-data-models/issues/5) +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a2...0.0.2a3) **Merged pull requests:** -- Formally define and document Access Roles values [\#6](https://github.com/NeonGeckoCom/neon-data-models/pull/6) ([NeonDaniel](https://github.com/NeonDaniel)) +- Define gradio context used by `neon-iris` [\#20](https://github.com/NeonGeckoCom/neon-data-models/pull/20) ([NeonDaniel](https://github.com/NeonDaniel)) -## [0.0.0a2](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a2) (2024-11-19) +## [0.0.2a2](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a2) (2025-04-04) -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.0a1...0.0.0a2) +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.2a1...0.0.2a2) **Merged pull requests:** -- Update TokenConfig and add JWT models [\#2](https://github.com/NeonGeckoCom/neon-data-models/pull/2) ([NeonDaniel](https://github.com/NeonDaniel)) +- Implement MQ models for chatbots API [\#19](https://github.com/NeonGeckoCom/neon-data-models/pull/19) ([NeonDaniel](https://github.com/NeonDaniel)) -## [0.0.0a1](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.0a1) (2024-11-04) +## [0.0.2a1](https://github.com/NeonGeckoCom/neon-data-models/tree/0.0.2a1) (2025-03-27) -[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/ccd1aff89a6ecb260cd95851a177eb0db02a86f3...0.0.0a1) +[Full Changelog](https://github.com/NeonGeckoCom/neon-data-models/compare/0.0.1...0.0.2a1) **Merged pull requests:** -- Initial Implementation [\#1](https://github.com/NeonGeckoCom/neon-data-models/pull/1) ([NeonDaniel](https://github.com/NeonDaniel)) +- Implement models for Neon MQ API [\#18](https://github.com/NeonGeckoCom/neon-data-models/pull/18) ([NeonDaniel](https://github.com/NeonDaniel))