Files
Mofox-Core/src/common/database/compatibility/adapter.py
Windpicker-owo 61de975d73 feat(database): 完成API层、Utils层和兼容层重构 (Stage 4-6)
Stage 4: API层重构
=================
新增文件:
- api/crud.py (430行): CRUDBase泛型类,提供12个CRUD方法
  * get, get_by, get_multi, create, update, delete
  * count, exists, get_or_create, bulk_create, bulk_update
  * 集成缓存: 自动缓存读操作,写操作清除缓存
  * 集成批处理: 可选use_batch参数透明使用AdaptiveBatchScheduler

- api/query.py (461行): 高级查询构建器
  * QueryBuilder: 链式调用,MongoDB风格操作符
    - 操作符: __gt, __lt, __gte, __lte, __ne, __in, __nin, __like, __isnull
    - 方法: filter, filter_or, order_by, limit, offset, no_cache
    - 执行: all, first, count, exists, paginate
  * AggregateQuery: 聚合查询
    - sum, avg, max, min, group_by_count

- api/specialized.py (461行): 业务特定API
  * ActionRecords: store_action_info, get_recent_actions
  * Messages: get_chat_history, get_message_count, save_message
  * PersonInfo: get_or_create_person, update_person_affinity
  * ChatStreams: get_or_create_chat_stream, get_active_streams
  * LLMUsage: record_llm_usage, get_usage_statistics
  * UserRelationships: get_user_relationship, update_relationship_affinity

- 更新api/__init__.py: 导出所有API接口

Stage 5: Utils层实现
===================
新增文件:
- utils/decorators.py (320行): 数据库操作装饰器
  * @retry: 自动重试失败操作,指数退避
  * @timeout: 超时控制
  * @cached: 自动缓存函数结果
  * @measure_time: 性能测量,慢查询日志
  * @transactional: 事务管理,自动提交/回滚
  * @db_operation: 组合装饰器

- utils/monitoring.py (330行): 性能监控系统
  * DatabaseMonitor: 单例监控器
  * OperationMetrics: 操作指标 (次数、时间、错误)
  * DatabaseMetrics: 全局指标
    - 连接池统计
    - 缓存命中率
    - 批处理统计
    - 预加载统计
  * 便捷函数: get_monitor, record_operation, print_stats

- 更新utils/__init__.py: 导出装饰器和监控函数

Stage 6: 兼容层实现
==================
新增目录: compatibility/
- adapter.py (370行): 向后兼容适配器
  * 完全兼容旧API签名: db_query, db_save, db_get, store_action_info
  * 支持MongoDB风格操作符 (\, \, \)
  * 内部使用新架构 (QueryBuilder + CRUDBase)
  * 保持返回dict格式不变
  * MODEL_MAPPING: 25个模型映射

- __init__.py: 导出兼容API

更新database/__init__.py:
- 导出核心层 (engine, session, models, migration)
- 导出优化层 (cache, preloader, batch_scheduler)
- 导出API层 (CRUD, Query, 业务API)
- 导出Utils层 (装饰器, 监控)
- 导出兼容层 (db_query, db_save等)

核心特性
========
 类型安全: Generic[T]提供完整类型推断
 缓存透明: 自动缓存,用户无需关心
 批处理透明: 可选批处理,自动优化高频写入
 链式查询: 流畅的API设计
 业务封装: 常用操作封装成便捷函数
 向后兼容: 兼容层保证现有代码无缝迁移
 性能监控: 完整的指标收集和报告

统计数据
========
- 新增文件: 7个
- 代码行数: ~2050行
- API函数: 14个业务API + 6个装饰器
- 兼容函数: 5个 (db_query, db_save, db_get等)

下一步
======
- 更新28个文件的import语句 (从sqlalchemy_database_api迁移)
- 移动旧文件到old/目录
- 编写Stage 4-6的测试
- 集成测试验证兼容性
2025-11-01 13:27:33 +08:00

362 lines
12 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

"""兼容层适配器
提供向后兼容的API将旧的数据库API调用转换为新架构的调用
保持原有函数签名和行为不变
"""
import time
from typing import Any, Optional
import orjson
from sqlalchemy import and_, asc, desc, select
from src.common.database.api import (
CRUDBase,
QueryBuilder,
store_action_info as new_store_action_info,
)
from src.common.database.core.models import (
ActionRecords,
CacheEntries,
ChatStreams,
Emoji,
Expression,
GraphEdges,
GraphNodes,
ImageDescriptions,
Images,
LLMUsage,
MaiZoneScheduleStatus,
Memory,
Messages,
OnlineTime,
PersonInfo,
PermissionNodes,
Schedule,
ThinkingLog,
UserPermissions,
UserRelationships,
)
from src.common.database.core.session import get_db_session
from src.common.logger import get_logger
logger = get_logger("database.compatibility")
# 模型映射表,用于通过名称获取模型类
MODEL_MAPPING = {
"Messages": Messages,
"ActionRecords": ActionRecords,
"PersonInfo": PersonInfo,
"ChatStreams": ChatStreams,
"LLMUsage": LLMUsage,
"Emoji": Emoji,
"Images": Images,
"ImageDescriptions": ImageDescriptions,
"OnlineTime": OnlineTime,
"Memory": Memory,
"Expression": Expression,
"ThinkingLog": ThinkingLog,
"GraphNodes": GraphNodes,
"GraphEdges": GraphEdges,
"Schedule": Schedule,
"MaiZoneScheduleStatus": MaiZoneScheduleStatus,
"CacheEntries": CacheEntries,
"UserRelationships": UserRelationships,
"PermissionNodes": PermissionNodes,
"UserPermissions": UserPermissions,
}
# 为每个模型创建CRUD实例
_crud_instances = {name: CRUDBase(model) for name, model in MODEL_MAPPING.items()}
async def build_filters(model_class, filters: dict[str, Any]):
"""构建查询过滤条件兼容MongoDB风格操作符
Args:
model_class: SQLAlchemy模型类
filters: 过滤条件字典
Returns:
条件列表
"""
conditions = []
for field_name, value in filters.items():
if not hasattr(model_class, field_name):
logger.warning(f"模型 {model_class.__name__} 中不存在字段 '{field_name}'")
continue
field = getattr(model_class, field_name)
if isinstance(value, dict):
# 处理 MongoDB 风格的操作符
for op, op_value in value.items():
if op == "$gt":
conditions.append(field > op_value)
elif op == "$lt":
conditions.append(field < op_value)
elif op == "$gte":
conditions.append(field >= op_value)
elif op == "$lte":
conditions.append(field <= op_value)
elif op == "$ne":
conditions.append(field != op_value)
elif op == "$in":
conditions.append(field.in_(op_value))
elif op == "$nin":
conditions.append(~field.in_(op_value))
else:
logger.warning(f"未知操作符 '{op}' (字段: '{field_name}')")
else:
# 直接相等比较
conditions.append(field == value)
return conditions
def _model_to_dict(instance) -> dict[str, Any]:
"""将模型实例转换为字典
Args:
instance: 模型实例
Returns:
字典表示
"""
if instance is None:
return None
result = {}
for column in instance.__table__.columns:
result[column.name] = getattr(instance, column.name)
return result
async def db_query(
model_class,
data: Optional[dict[str, Any]] = None,
query_type: Optional[str] = "get",
filters: Optional[dict[str, Any]] = None,
limit: Optional[int] = None,
order_by: Optional[list[str]] = None,
single_result: Optional[bool] = False,
) -> list[dict[str, Any]] | dict[str, Any] | None:
"""执行异步数据库查询操作兼容旧API
Args:
model_class: SQLAlchemy模型类
data: 用于创建或更新的数据字典
query_type: 查询类型 ("get", "create", "update", "delete", "count")
filters: 过滤条件字典
limit: 限制结果数量
order_by: 排序字段,前缀'-'表示降序
single_result: 是否只返回单个结果
Returns:
根据查询类型返回相应结果
"""
try:
if query_type not in ["get", "create", "update", "delete", "count"]:
raise ValueError("query_type must be 'get', 'create', 'update', 'delete' or 'count'")
# 获取CRUD实例
model_name = model_class.__name__
crud = _crud_instances.get(model_name)
if not crud:
crud = CRUDBase(model_class)
if query_type == "get":
# 使用QueryBuilder
query_builder = QueryBuilder(model_class)
# 应用过滤条件
if filters:
# 将MongoDB风格过滤器转换为QueryBuilder格式
for field_name, value in filters.items():
if isinstance(value, dict):
for op, op_value in value.items():
if op == "$gt":
query_builder = query_builder.filter(**{f"{field_name}__gt": op_value})
elif op == "$lt":
query_builder = query_builder.filter(**{f"{field_name}__lt": op_value})
elif op == "$gte":
query_builder = query_builder.filter(**{f"{field_name}__gte": op_value})
elif op == "$lte":
query_builder = query_builder.filter(**{f"{field_name}__lte": op_value})
elif op == "$ne":
query_builder = query_builder.filter(**{f"{field_name}__ne": op_value})
elif op == "$in":
query_builder = query_builder.filter(**{f"{field_name}__in": op_value})
elif op == "$nin":
query_builder = query_builder.filter(**{f"{field_name}__nin": op_value})
else:
query_builder = query_builder.filter(**{field_name: value})
# 应用排序
if order_by:
query_builder = query_builder.order_by(*order_by)
# 应用限制
if limit:
query_builder = query_builder.limit(limit)
# 执行查询
if single_result:
result = await query_builder.first()
return _model_to_dict(result)
else:
results = await query_builder.all()
return [_model_to_dict(r) for r in results]
elif query_type == "create":
if not data:
logger.error("创建操作需要提供data参数")
return None
instance = await crud.create(data)
return _model_to_dict(instance)
elif query_type == "update":
if not filters or not data:
logger.error("更新操作需要提供filters和data参数")
return None
# 先查找记录
query_builder = QueryBuilder(model_class)
for field_name, value in filters.items():
query_builder = query_builder.filter(**{field_name: value})
instance = await query_builder.first()
if not instance:
logger.warning(f"未找到匹配的记录: {filters}")
return None
# 更新记录
updated = await crud.update(instance.id, data)
return _model_to_dict(updated)
elif query_type == "delete":
if not filters:
logger.error("删除操作需要提供filters参数")
return None
# 先查找记录
query_builder = QueryBuilder(model_class)
for field_name, value in filters.items():
query_builder = query_builder.filter(**{field_name: value})
instance = await query_builder.first()
if not instance:
logger.warning(f"未找到匹配的记录: {filters}")
return None
# 删除记录
success = await crud.delete(instance.id)
return {"deleted": success}
elif query_type == "count":
query_builder = QueryBuilder(model_class)
# 应用过滤条件
if filters:
for field_name, value in filters.items():
query_builder = query_builder.filter(**{field_name: value})
count = await query_builder.count()
return {"count": count}
except Exception as e:
logger.error(f"数据库操作失败: {e}", exc_info=True)
return None if single_result or query_type != "get" else []
async def db_save(
model_class,
data: dict[str, Any],
key_field: str,
key_value: Any,
) -> Optional[dict[str, Any]]:
"""保存或更新记录兼容旧API
Args:
model_class: SQLAlchemy模型类
data: 数据字典
key_field: 主键字段名
key_value: 主键值
Returns:
保存的记录数据或None
"""
try:
model_name = model_class.__name__
crud = _crud_instances.get(model_name)
if not crud:
crud = CRUDBase(model_class)
# 使用get_or_create
instance = await crud.get_or_create(
defaults=data,
**{key_field: key_value},
)
return _model_to_dict(instance)
except Exception as e:
logger.error(f"保存数据库记录出错: {e}", exc_info=True)
return None
async def db_get(
model_class,
filters: Optional[dict[str, Any]] = None,
limit: Optional[int] = None,
order_by: Optional[str] = None,
single_result: Optional[bool] = False,
) -> list[dict[str, Any]] | dict[str, Any] | None:
"""从数据库获取记录兼容旧API
Args:
model_class: SQLAlchemy模型类
filters: 过滤条件
limit: 结果数量限制
order_by: 排序字段,前缀'-'表示降序
single_result: 是否只返回单个结果
Returns:
记录数据或None
"""
order_by_list = [order_by] if order_by else None
return await db_query(
model_class=model_class,
query_type="get",
filters=filters,
limit=limit,
order_by=order_by_list,
single_result=single_result,
)
async def store_action_info(
chat_stream=None,
action_build_into_prompt: bool = False,
action_prompt_display: str = "",
action_done: bool = True,
thinking_id: str = "",
action_data: Optional[dict] = None,
action_name: str = "",
) -> Optional[dict[str, Any]]:
"""存储动作信息到数据库兼容旧API
直接使用新的specialized API
"""
return await new_store_action_info(
chat_stream=chat_stream,
action_build_into_prompt=action_build_into_prompt,
action_prompt_display=action_prompt_display,
action_done=action_done,
thinking_id=thinking_id,
action_data=action_data,
action_name=action_name,
)