feat: 重构渠道模型与管道架构,全面升级 UI 主题和报表功能
核心重构 — 去除 channel 字段,引入 sourceAccount 模型: - 从 ImportedEvent、Rule、EnhancedRule、OcrRule 等接口中彻底移除 channel 字段 - rules.ts 中 resolveChannelAccount → resolveSourceAccount,规则匹配与账户解析不再依赖渠道概念 - dedup.ts 重写去重逻辑:从基于渠道匹配改为基于交易对手(counterparty)匹配,支持相同金额/交易对手/时间窗口的多级置信度判断 - transferRecognizer.ts 增加资产负债表账户校验,确保转账双方均为 Assets/Liabilities 类账户 - 全局替换影响:types、rules、ocr、adapters、adapters-migrations、所有服务层和测试 新增基础设施: - domain/constants.ts — 统一常量定义(支付包名、截图关键词、去重参数、方向检测函数 detectDirection()),消除 OCR/SMS/截图等模块的重复定义 - domain/channelConfig.ts — 渠道配置系统(支付宝/微信/银行),支持按包名和名称查找 - domain/pipelineSingleton.ts — 共享 BillPipeline 单例,解决 importStore/automationStore 的互斥锁共享问题 - domain/transactionBuilder.ts — 统一交易构建入口 buildAndSaveTransaction(),同时服务手动录入和无障碍监听 OCR 增强: - 新增账单详情页解析(parseDetailPageBill),支持支付宝/微信详情页结构化提取 - checkIsDetailPage() 识别详情页特征词,防止误提取(如"消费1次"被误读为金额) - 金额正则支持千分位逗号分隔,商户名正则改用 lookahead 边界匹配 - 时间解析支持中文格式(年月日)和跨年推断 - OcrProcessor 新增详情页路由,跳过 Layer 1 规则匹配 UI 全面升级: - 主题重设计:accent 色从绿色改为靛蓝(#4F46E5),深色模式适配 OLED 纯黑,引入 Quicksand/Caveat 字体 - 新增 commonStyles.ts 统一 chip/input/modal 等通用样式 - 首页 Bento 网格布局:净资产英雄卡片 + 定期账单/月度统计并排展示 - 报表新增周报标签页,月报整合日历视图(支持点击查看当日交易明细) - TrendLine 图表从 View 条形图重写为 SVG 贝塞尔曲线 - CategoryPicker 从水平滚动改为 4 列网格 + emoji 图标 - Button/Card 增加 press 缩放动画 管道与自动化改进: - automationPipeline.ts 新增 handleIncomingBillEvent() 实时账单处理(悬浮账单卡片 + 前台 Alert 确认) - 新增无障碍文本直解析 parseAndProcessAccessibilityTexts(),微信/支付宝详情页绕过 OCR - rules.ts 新增智能还款检测(花呗/信用卡还款自动路由)和退款视为收入处理 - metadataStore 默认规则精简为 6 条通用规则,移除约 20 条个人化硬编码规则 存储与同步: - storePersistence.ts 原子写入 + 崩溃恢复 + 重试机制 - 备份升级到 v2 格式,包含 settings 和 metadata - 同步路径统一从 mobile.bean 改为 main.bean - _layout.tsx 启动时自动迁移旧 mobile.bean 到 main.bean 其他: - 删除独立日历页面,功能合并到报表月报标签 - i18n 清理:移除渠道相关翻译,新增 50+ 翻译键 - docs/android-build-guide.md 重写为 APK 体积优化指南 - 新增 design-system/beancount-mobile/MASTER.md 设计系统文档 - 测试全面更新覆盖以上所有变更
This commit is contained in:
+11
-5
@@ -14,8 +14,8 @@ plugins/ppocr/
|
||||
│ └── OcrPackage.kt # RN Package 注册(注入到 MainApplication.getPackages)
|
||||
└── assets/ # ONNX 模型 + 字典(需自行下载放置)
|
||||
├── ppocrv5_det.onnx # 文本检测模型
|
||||
├── ppocrv5_rec.onnx # 文本识别模型
|
||||
└── ppocr_keys_v1.txt # CJK 字典(CTC 解码用)
|
||||
├── ppocrv5_rec.onnx # 文本识别模型(多语言,输出 18385 维)
|
||||
└── ppocrv5_dict.txt # PP-OCRv5 多语言字典(18383 字符,CTC 解码用)
|
||||
```
|
||||
|
||||
## 模型获取(一键下载)
|
||||
@@ -33,12 +33,16 @@ curl -L -o ppocrv5_det.onnx https://huggingface.co/ilaylow/PP_OCRv5_mobile_onnx/
|
||||
# rec 模型(16.6 MB)
|
||||
curl -L -o ppocrv5_rec.onnx https://huggingface.co/ilaylow/PP_OCRv5_mobile_onnx/resolve/main/ppocrv5_rec.onnx
|
||||
|
||||
# CJK 字典(26 KB,PaddleOCR 标准字典)
|
||||
curl -L -o ppocr_keys_v1.txt https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/release/2.6/ppocr/utils/ppocr_keys_v1.txt
|
||||
# PP-OCRv5 多语言字典(74 KB,必须与上面的 rec 模型配套)
|
||||
curl -L -o ppocrv5_dict.txt https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/main/ppocr/utils/dict/ppocrv5_dict.txt
|
||||
```
|
||||
|
||||
或用 HuggingFace CLI(首次下载原生模型再转 ONNX 的方式,参见历史 git log)。
|
||||
|
||||
> ⚠️ **字典必须与 rec 模型配套**:ppocrv5_rec.onnx 输出 18385 维(= 18383 字符 + blank + 特殊位),
|
||||
> 必须使用 `ppocrv5_dict.txt`(18383 行)。若错用旧版 `ppocr_keys_v1.txt`(仅 6623 行),
|
||||
> CTC 解码会把真实字符的高索引全部丢弃,只输出形如 `'消'青'露'仰'` 的单引号穿插单字符乱码。
|
||||
|
||||
> 来源说明:[ilaylow/PP_OCRv5_mobile_onnx](https://huggingface.co/ilaylow/PP_OCRv5_mobile_onnx) 是社区维护的 PP-OCRv5 mobile ONNX 镜像,基于官方 [PaddlePaddle/PP-OCRv5_mobile_det](https://huggingface.co/PaddlePaddle/PP-OCRv5_mobile_det) 与 [_rec](https://huggingface.co/PaddlePaddle/PP-OCRv5_mobile_rec) 转换而来。
|
||||
|
||||
## 性能配置(参考 AutoAccounting OcrProcessor.kt)
|
||||
@@ -75,4 +79,6 @@ JS 层通过 `src/services/ocrBridge.ts` 的 `NativeOcrBridge` 调用,桥接
|
||||
- `android/OcrPackage.kt`:✅ RN Package 注册
|
||||
- `assets/`:需自行下载放置(见上「模型获取」),版权/体积原因不入仓库
|
||||
|
||||
真机构建步骤:放置模型文件 → `npx expo prebuild --platform android` → `npx expo run:android`。
|
||||
真机构建步骤:放置模型文件 → `npx expo prebuild --platform android`(Config Plugin 会把 Kotlin 源码与 `assets/` 下的模型/字典复制进 `android/`)→ `npx expo run:android`。
|
||||
|
||||
> 若之前已 prebuild 过且更换过字典/模型文件,务必重新执行 `npx expo prebuild --clean`,否则 `android/app/src/main/assets/` 下可能残留旧字典(如 `ppocr_keys_v1.txt`),导致新代码找不到配套字典。
|
||||
|
||||
@@ -0,0 +1,659 @@
|
||||
package com.beancount.mobile.ppocr
|
||||
|
||||
import android.graphics.Bitmap
|
||||
import android.graphics.BitmapFactory
|
||||
import android.util.Base64
|
||||
import android.util.Log
|
||||
import ai.onnxruntime.OnnxTensor
|
||||
import ai.onnxruntime.OrtEnvironment
|
||||
import ai.onnxruntime.OrtSession
|
||||
import com.facebook.react.bridge.Arguments
|
||||
import com.facebook.react.bridge.Promise
|
||||
import com.facebook.react.bridge.ReactApplicationContext
|
||||
import com.facebook.react.bridge.ReactContextBaseJavaModule
|
||||
import com.facebook.react.bridge.ReactMethod
|
||||
import com.facebook.react.bridge.ReadableArray
|
||||
import com.facebook.react.bridge.WritableMap
|
||||
import com.facebook.react.bridge.WritableNativeArray
|
||||
import com.facebook.react.bridge.WritableNativeMap
|
||||
import com.facebook.react.module.annotations.ReactModule
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.SupervisorJob
|
||||
import kotlinx.coroutines.launch
|
||||
import java.io.BufferedReader
|
||||
import java.io.InputStreamReader
|
||||
import java.nio.FloatBuffer
|
||||
import java.util.concurrent.locks.ReentrantLock
|
||||
import kotlin.math.max
|
||||
import kotlin.math.min
|
||||
|
||||
/**
|
||||
* PP-OCRv5 (ONNX Runtime) React Native Bridge(plan.md「3.4 Layer 2」+「决策 4 Config Plugin」)。
|
||||
*
|
||||
* 引擎:ONNX Runtime(跨平台、微软官方、Windows 友好),替代 NCNN 路线。
|
||||
* 模型:ppocrv5_det.onnx + ppocrv5_rec.onnx(从 ilaylow/PP_OCRv5_mobile_onnx 下载)。
|
||||
* 字典:ppocrv5_dict.txt(PP-OCRv5 多语言字典,18383 字符;rec 模型 18385 维输出 = 字典 + blank + 特殊位)。
|
||||
*
|
||||
* 流水线:
|
||||
* 1. det(文本检测):bitmap → DB 后处理得到文本框
|
||||
* 2. rec(文本识别):每个框 crop → resize 到 48px 高 → CTC 解码
|
||||
*
|
||||
* 性能优化(参考 AutoAccounting OcrProcessor.kt):
|
||||
* - 短边压缩到 720px(像素量比 1440p 减少约 75%)
|
||||
* - CPU 执行(兼容性最稳,部分设备 GPU 会崩溃)
|
||||
* - det 最大边限制 960(PaddleOCR 默认 limit_max_side_len)
|
||||
*
|
||||
* JS 层通过 NativeModules.PpOcr.recognizeText(base64) 调用。
|
||||
*/
|
||||
const val OCR_MODULE_NAME = "PpOcr"
|
||||
|
||||
@ReactModule(name = OCR_MODULE_NAME)
|
||||
class OcrModule(private val context: ReactApplicationContext) :
|
||||
ReactContextBaseJavaModule(context) {
|
||||
|
||||
private val scope = CoroutineScope(SupervisorJob() + Dispatchers.IO)
|
||||
private val lock = ReentrantLock()
|
||||
|
||||
private var ortEnv: OrtEnvironment? = null
|
||||
private var detSession: OrtSession? = null
|
||||
private var recSession: OrtSession? = null
|
||||
private var dictionary: List<String> = emptyList()
|
||||
@Volatile private var initialized = false
|
||||
@Volatile private var initFailed = false
|
||||
|
||||
override fun getName(): String = OCR_MODULE_NAME
|
||||
|
||||
override fun initialize() {
|
||||
super.initialize()
|
||||
// 异步加载模型,避免阻塞 RN 桥初始化
|
||||
scope.launch { initEngine() }
|
||||
}
|
||||
|
||||
/** 从 assets 加载 det/rec ONNX 模型与字典。 */
|
||||
private fun initEngine() {
|
||||
lock.lock()
|
||||
try {
|
||||
if (initialized || initFailed) return
|
||||
val env = OrtEnvironment.getEnvironment()
|
||||
val opts = OrtSession.SessionOptions().apply {
|
||||
// CPU 线程数:2 是兼容性/性能的稳妥折中(高端机可调高)
|
||||
setInterOpNumThreads(2)
|
||||
setIntraOpNumThreads(2)
|
||||
// 移动端关闭内存优化里的图优化级别过高(部分模型会崩)
|
||||
setOptimizationLevel(OrtSession.SessionOptions.OptLevel.BASIC_OPT)
|
||||
}
|
||||
val detBytes = context.assets.open(ASSET_DET_MODEL).use { it.readBytes() }
|
||||
val recBytes = context.assets.open(ASSET_REC_MODEL).use { it.readBytes() }
|
||||
val det = env.createSession(detBytes, opts)
|
||||
val rec = env.createSession(recBytes, opts)
|
||||
val dict = loadDictionary()
|
||||
|
||||
detSession = det
|
||||
recSession = rec
|
||||
ortEnv = env
|
||||
dictionary = dict
|
||||
initialized = true
|
||||
Log.i(OCR_MODULE_NAME, "PP-OCRv5 ONNX 模型加载成功(det+rec, dict=${dict.size})")
|
||||
} catch (e: Exception) {
|
||||
initFailed = true
|
||||
Log.e(OCR_MODULE_NAME, "OCR 初始化失败: ${e.message}", e)
|
||||
Log.e(OCR_MODULE_NAME, "请确认 assets 下存在 $ASSET_DET_MODEL / $ASSET_REC_MODEL / $ASSET_DICT")
|
||||
} finally {
|
||||
lock.unlock()
|
||||
}
|
||||
}
|
||||
|
||||
/** rec 推理用的 OrtEnvironment(复用 ortEnv 单例)。 */
|
||||
private val recEnv: OrtEnvironment? get() = ortEnv
|
||||
|
||||
/**
|
||||
* 加载 ppocrv5_dict.txt 字典。
|
||||
*
|
||||
* PaddleOCR CTC 约定:模型输出 logits 的 index 0 是 blank,字符从 index 1 开始;
|
||||
* 字典条目 dictionary[i] 对应模型输出 index i+1。解码时 dictIdx = argmaxIdx - 1。
|
||||
* 字典本身不含 blank,运行时固定用 index 0 作 blank(见 ctcGreedyDecode)。
|
||||
*/
|
||||
private fun loadDictionary(): List<String> {
|
||||
val words = mutableListOf<String>()
|
||||
context.assets.open(ASSET_DICT).use { stream ->
|
||||
BufferedReader(InputStreamReader(stream, Charsets.UTF_8)).useLines { lines ->
|
||||
lines.forEach { line ->
|
||||
// PaddleOCR 字典每行一个字符(去掉行尾换行)
|
||||
words.add(line.trimEnd('\r', '\n'))
|
||||
}
|
||||
}
|
||||
}
|
||||
return words
|
||||
}
|
||||
|
||||
/**
|
||||
* 识别图片文本。
|
||||
* @param imageBase64 base64 编码的图片(JPEG/PNG)
|
||||
* @return 识别出的纯文本(所有行用 \n 连接)
|
||||
*/
|
||||
@ReactMethod
|
||||
fun recognizeText(imageBase64: String, promise: Promise) {
|
||||
scope.launch {
|
||||
var bitmap: Bitmap? = null
|
||||
var scaled: Bitmap? = null
|
||||
try {
|
||||
ensureReady()
|
||||
bitmap = decodeBase64(imageBase64)
|
||||
if (bitmap == null) {
|
||||
promise.reject("DECODE_FAILED", "base64 解码失败")
|
||||
return@launch
|
||||
}
|
||||
scaled = scaleDownForOcr(bitmap, OCR_MAX_SHORT_EDGE)
|
||||
val blocks = runInference(scaled)
|
||||
val text = blocks.joinToString("\n") { it.text }
|
||||
promise.resolve(text)
|
||||
} catch (e: Exception) {
|
||||
Log.e(OCR_MODULE_NAME, "recognizeText 异常: ${e.message}", e)
|
||||
promise.reject("OCR_ERROR", e.message)
|
||||
} finally {
|
||||
bitmap?.recycle()
|
||||
if (scaled !== bitmap) {
|
||||
scaled?.recycle()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 识别并返回带坐标的文本块(用于复杂版面)。
|
||||
* @return JSON 数组字符串:[{text, x, y, width, height, confidence}]
|
||||
*/
|
||||
@ReactMethod
|
||||
fun recognizeTextBlocks(imageBase64: String, promise: Promise) {
|
||||
scope.launch {
|
||||
var bitmap: Bitmap? = null
|
||||
var scaled: Bitmap? = null
|
||||
try {
|
||||
ensureReady()
|
||||
bitmap = decodeBase64(imageBase64)
|
||||
if (bitmap == null) {
|
||||
promise.reject("DECODE_FAILED", "base64 解码失败")
|
||||
return@launch
|
||||
}
|
||||
scaled = scaleDownForOcr(bitmap, OCR_MAX_SHORT_EDGE)
|
||||
val blocks = runInference(scaled)
|
||||
// 序列化为 RN WritableArray
|
||||
val arr = WritableNativeArray()
|
||||
for (b in blocks) {
|
||||
val map: WritableMap = WritableNativeMap()
|
||||
map.putString("text", b.text)
|
||||
map.putDouble("x", b.x.toDouble())
|
||||
map.putDouble("y", b.y.toDouble())
|
||||
map.putDouble("width", b.width.toDouble())
|
||||
map.putDouble("height", b.height.toDouble())
|
||||
map.putDouble("confidence", b.confidence.toDouble())
|
||||
arr.pushMap(map)
|
||||
}
|
||||
promise.resolve(arr)
|
||||
} catch (e: Exception) {
|
||||
Log.e(OCR_MODULE_NAME, "recognizeTextBlocks 异常: ${e.message}", e)
|
||||
promise.reject("OCR_ERROR", e.message)
|
||||
} finally {
|
||||
bitmap?.recycle()
|
||||
if (scaled !== bitmap) {
|
||||
scaled?.recycle()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** 引擎是否已就绪(模型加载完成)。 */
|
||||
@ReactMethod
|
||||
fun isReady(promise: Promise) {
|
||||
promise.resolve(initialized)
|
||||
}
|
||||
|
||||
// ============== 推理流水线 ==============
|
||||
|
||||
private fun ensureReady() {
|
||||
if (!initialized && !initFailed) initEngine()
|
||||
if (!initialized) throw IllegalStateException("OCR 引擎未就绪(模型未加载,${if (initFailed) "初始化失败" else "加载中"})")
|
||||
}
|
||||
|
||||
/** 完整推理:det 检测文本框 → 对每个框 rec 识别 → 返回带坐标的文本块。 */
|
||||
private fun runInference(bitmap: Bitmap): List<OcrBlock> {
|
||||
Log.i(OCR_MODULE_NAME, "runInference 开始: bitmap 尺寸 = ${bitmap.width}x${bitmap.height}")
|
||||
val det = detSession ?: run {
|
||||
Log.e(OCR_MODULE_NAME, "detSession 为空,放弃推理")
|
||||
return emptyList()
|
||||
}
|
||||
val rec = recSession ?: run {
|
||||
Log.e(OCR_MODULE_NAME, "recSession 为空,放弃推理")
|
||||
return emptyList()
|
||||
}
|
||||
|
||||
var resized: Bitmap? = null
|
||||
var detInputTensor: OnnxTensor? = null
|
||||
var detOutputs: OrtSession.Result? = null
|
||||
val results = mutableListOf<OcrBlock>()
|
||||
|
||||
try {
|
||||
// ---- 1. 文本检测(DB)----
|
||||
resized = resizeForDet(bitmap, DET_LIMIT_MAX_SIDE)
|
||||
Log.i(OCR_MODULE_NAME, "det 图像缩放后尺寸 = ${resized.width}x${resized.height}")
|
||||
val ratioX = bitmap.width.toFloat() / resized.width
|
||||
val ratioY = bitmap.height.toFloat() / resized.height
|
||||
|
||||
val detInput = preprocessDet(resized)
|
||||
detInputTensor = OnnxTensor.createTensor(recEnv, FloatBuffer.wrap(detInput.data), longArrayOf(1L, 3L, detInput.h.toLong(), detInput.w.toLong()))
|
||||
val detInputs = mapOf("x" to detInputTensor)
|
||||
detOutputs = det.run(detInputs)
|
||||
@Suppress("UNCHECKED_CAST")
|
||||
val detProb = (detOutputs[0].value as Array<Array<Array<FloatArray>>>)[0][0] // [H,W]
|
||||
Log.i(OCR_MODULE_NAME, "det 推理完成,概率图尺寸 = ${detProb.size}x${detProb[0].size}")
|
||||
|
||||
// DB 后处理:threshold → 轮廓 → 最小外接矩形
|
||||
val boxes = dbPostprocess(detProb, detInput.h, detInput.w, ratioX, ratioY)
|
||||
Log.i(OCR_MODULE_NAME, "dbPostprocess 后处理完成,检测到文本框数量 = ${boxes.size}")
|
||||
|
||||
if (boxes.isEmpty()) return emptyList()
|
||||
|
||||
// ---- 2. 文本识别(CRNN+CTC)----
|
||||
for ((idx, box) in boxes.withIndex()) {
|
||||
var crop: Bitmap? = null
|
||||
var recInputTensor: OnnxTensor? = null
|
||||
var recOutputs: OrtSession.Result? = null
|
||||
try {
|
||||
crop = cropBox(bitmap, box)
|
||||
if (crop == null) {
|
||||
Log.w(OCR_MODULE_NAME, "裁剪文本框失败 (index = $idx)")
|
||||
continue
|
||||
}
|
||||
val recInput = preprocessRec(crop)
|
||||
recInputTensor = OnnxTensor.createTensor(recEnv, FloatBuffer.wrap(recInput.data), longArrayOf(1L, 3L, REC_IMAGE_HEIGHT.toLong(), recInput.w.toLong()))
|
||||
val recInputs = mapOf("x" to recInputTensor)
|
||||
recOutputs = rec.run(recInputs)
|
||||
// 输出 shape: [1, T, numClasses]
|
||||
@Suppress("UNCHECKED_CAST")
|
||||
val logits = (recOutputs[0].value as Array<Array<FloatArray>>)[0]
|
||||
|
||||
val (text, conf) = ctcGreedyDecode(logits)
|
||||
val xs = box.map { it[0] }
|
||||
val ys = box.map { it[1] }
|
||||
val minX = (xs.minOrNull() ?: 0f).toInt()
|
||||
val minY = (ys.minOrNull() ?: 0f).toInt()
|
||||
val maxX = (xs.maxOrNull() ?: 0f).toInt()
|
||||
val maxY = (ys.maxOrNull() ?: 0f).toInt()
|
||||
val w = maxX - minX
|
||||
val h = maxY - minY
|
||||
Log.i(OCR_MODULE_NAME, "文本框 $idx 识别结果 = '$text', 坐标 = ($minX, $minY, $w, $h), 置信度 = $conf")
|
||||
if (text.isNotEmpty()) {
|
||||
results.add(OcrBlock(text, minX.toFloat(), minY.toFloat(), w.toFloat(), h.toFloat(), conf))
|
||||
}
|
||||
} finally {
|
||||
crop?.recycle()
|
||||
recInputTensor?.close()
|
||||
recOutputs?.close()
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
if (resized !== bitmap) {
|
||||
resized?.recycle()
|
||||
}
|
||||
detInputTensor?.close()
|
||||
detOutputs?.close()
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
// ============== 前处理 ==============
|
||||
|
||||
/** det 前处理:resize → BCHW → normalize(mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225])。 */
|
||||
private fun preprocessDet(bmp: Bitmap): TensorData {
|
||||
val w = bmp.width
|
||||
val h = bmp.height
|
||||
val pixels = IntArray(w * h)
|
||||
bmp.getPixels(pixels, 0, w, 0, 0, w, h)
|
||||
val data = FloatArray(3 * w * h)
|
||||
// BCHW 顺序
|
||||
val means = floatArrayOf(0.485f, 0.456f, 0.406f)
|
||||
val stds = floatArrayOf(0.229f, 0.224f, 0.225f)
|
||||
for (c in 0..2) {
|
||||
val mean = means[c]
|
||||
val std = stds[c]
|
||||
for (y in 0 until h) {
|
||||
for (x in 0 until w) {
|
||||
val px = pixels[y * w + x]
|
||||
// 提取 R/G/B(c=0→R, 1→G, 2→B)
|
||||
val channelVal = (px shr (16 - 8 * c)) and 0xFF
|
||||
data[c * w * h + y * w + x] = (channelVal / 255.0f - mean) / std
|
||||
}
|
||||
}
|
||||
}
|
||||
return TensorData(data, w, h)
|
||||
}
|
||||
|
||||
/** rec 前处理:crop → resize 到 48 高(保持宽高比)→ pad 到整除 4 → normalize。 */
|
||||
private fun preprocessRec(bmp: Bitmap): TensorData {
|
||||
var w = bmp.width
|
||||
val h = bmp.height
|
||||
// resize 到高度 48,宽度等比缩放
|
||||
var resizedW = (w.toFloat() / h * REC_IMAGE_HEIGHT).toInt()
|
||||
// 宽度上限,避免单行过长爆显存
|
||||
resizedW = min(resizedW, REC_MAX_WIDTH)
|
||||
resizedW = max(resizedW, 1)
|
||||
val resized = if (resizedW == w && h == REC_IMAGE_HEIGHT) bmp
|
||||
else Bitmap.createScaledBitmap(bmp, resizedW, REC_IMAGE_HEIGHT, true)
|
||||
w = resized.width
|
||||
val pixels = IntArray(w * REC_IMAGE_HEIGHT)
|
||||
resized.getPixels(pixels, 0, w, 0, 0, w, REC_IMAGE_HEIGHT)
|
||||
val data = FloatArray(3 * w * REC_IMAGE_HEIGHT)
|
||||
val means = floatArrayOf(0.5f, 0.5f, 0.5f)
|
||||
val stds = floatArrayOf(0.5f, 0.5f, 0.5f)
|
||||
for (c in 0..2) {
|
||||
val mean = means[c]
|
||||
val std = stds[c]
|
||||
for (y in 0 until REC_IMAGE_HEIGHT) {
|
||||
for (x in 0 until w) {
|
||||
val px = pixels[y * w + x]
|
||||
val channelVal = (px shr (16 - 8 * c)) and 0xFF
|
||||
data[c * w * REC_IMAGE_HEIGHT + y * w + x] = (channelVal / 255.0f - mean) / std
|
||||
}
|
||||
}
|
||||
}
|
||||
if (resized !== bmp) resized.recycle()
|
||||
return TensorData(data, w, REC_IMAGE_HEIGHT)
|
||||
}
|
||||
|
||||
// ============== DB 后处理(简化版) ==============
|
||||
// 参考 PaddleOCR db_postprocess:sigmoid → threshold → 连通域 → 最小外接矩形
|
||||
// 这里用轻量实现:逐像素阈值化后用投影法估框,对常见单/多行账单足够。
|
||||
|
||||
/**
|
||||
* DB 后处理:sigmoid + 阈值 0.3 → 二值图 → 连通域外接矩形。
|
||||
* 简化版:用水平投影切行 + 垂直投影切列,得到矩形框(对账单类版面够用)。
|
||||
*/
|
||||
private fun dbPostprocess(prob: Array<FloatArray>, h: Int, w: Int, ratioX: Float, ratioY: Float): List<List<FloatArray>> {
|
||||
// 自动检测是否需要 Sigmoid
|
||||
var minVal = Float.MAX_VALUE
|
||||
var maxVal = Float.MIN_VALUE
|
||||
for (y in 0 until h) {
|
||||
for (x in 0 until w) {
|
||||
val v = prob[y][x]
|
||||
if (v < minVal) minVal = v
|
||||
if (v > maxVal) maxVal = v
|
||||
}
|
||||
}
|
||||
val needSigmoid = minVal < -0.05f || maxVal > 1.05f
|
||||
Log.i(OCR_MODULE_NAME, "DB prob min=$minVal, max=$maxVal, needSigmoid=$needSigmoid")
|
||||
|
||||
// 过滤状态栏(前8%)和导航栏(后8%),避免其上的干扰字符(如电池、时间、返回键)影响识别或导致行粘连
|
||||
val startY = (h * 0.08).toInt()
|
||||
val endY = (h * 0.92).toInt()
|
||||
|
||||
val binMask = Array(h) { IntArray(w) }
|
||||
var activeCountTotal = 0
|
||||
for (y in 0 until h) {
|
||||
for (x in 0 until w) {
|
||||
if (y < startY || y > endY) {
|
||||
binMask[y][x] = 0
|
||||
continue
|
||||
}
|
||||
val raw = prob[y][x]
|
||||
val sig = if (needSigmoid) {
|
||||
1.0f / (1.0f + Math.exp(-raw.toDouble()).toFloat())
|
||||
} else {
|
||||
raw
|
||||
}
|
||||
val isActive = if (sig > DET_THRESH) 1 else 0
|
||||
binMask[y][x] = isActive
|
||||
if (isActive == 1) activeCountTotal++
|
||||
}
|
||||
}
|
||||
Log.i(OCR_MODULE_NAME, "二值化完成: 活跃像素 = $activeCountTotal / ${w * h}")
|
||||
|
||||
// 清理垂直干扰线(如滚动条、背景边框线):如果某列在文本有效区域内的活跃像素超过该区域高度的 30%,视为干扰列,整列清零
|
||||
val maxColActive = ((endY - startY) * 0.3).toInt()
|
||||
var clearedColsCount = 0
|
||||
for (x in 0 until w) {
|
||||
var colActive = 0
|
||||
for (y in startY..endY) {
|
||||
if (binMask[y][x] == 1) colActive++
|
||||
}
|
||||
if (colActive > maxColActive) {
|
||||
clearedColsCount++
|
||||
for (y in 0 until h) {
|
||||
binMask[y][x] = 0
|
||||
}
|
||||
}
|
||||
}
|
||||
Log.i(OCR_MODULE_NAME, "垂直线噪清理完成: 清理了 $clearedColsCount / $w 列")
|
||||
|
||||
// 水平投影:按行找文本行
|
||||
val rowHits = IntArray(h)
|
||||
for (y in 0 until h) {
|
||||
var sum = 0
|
||||
for (x in 0 until w) sum += binMask[y][x]
|
||||
rowHits[y] = sum
|
||||
}
|
||||
val minRowWidth = max(1, w / 20) // 一行至少要有这么多像素才算文本
|
||||
val rowRanges = mutableListOf<IntArray>()
|
||||
var inLine = false
|
||||
var lineStart = 0
|
||||
for (y in 0 until h) {
|
||||
val isText = rowHits[y] >= minRowWidth
|
||||
if (isText && !inLine) { inLine = true; lineStart = y }
|
||||
else if (!isText && inLine) {
|
||||
rowRanges.add(intArrayOf(lineStart, y - 1))
|
||||
inLine = false
|
||||
}
|
||||
}
|
||||
if (inLine) rowRanges.add(intArrayOf(lineStart, h - 1))
|
||||
Log.i(OCR_MODULE_NAME, "dbPostprocess 水平分割完成,找到行Ranges数 = ${rowRanges.size}")
|
||||
|
||||
val boxes = mutableListOf<List<FloatArray>>()
|
||||
// 对每行做垂直投影切列(账单每行通常是连续一段或多段)
|
||||
for ((y0, y1) in rowRanges.map { it[0] to it[1] }) {
|
||||
val colHits = IntArray(w)
|
||||
for (x in 0 until w) {
|
||||
var sum = 0
|
||||
for (y in y0..y1) sum += binMask[y][x]
|
||||
colHits[x] = sum
|
||||
}
|
||||
val minColHeight = max(1, (y1 - y0 + 1) / 12)
|
||||
var inSeg = false
|
||||
var segStart = 0
|
||||
var segs = mutableListOf<IntArray>()
|
||||
for (x in 0 until w) {
|
||||
val isText = colHits[x] >= minColHeight
|
||||
if (isText && !inSeg) { inSeg = true; segStart = x }
|
||||
else if (!isText && inSeg) {
|
||||
// 合并间隔很近的段
|
||||
if (segs.isNotEmpty() && segStart - segs.last()[1] < DET_MERGE_GAP) {
|
||||
segs.last()[1] = x - 1
|
||||
} else {
|
||||
segs.add(intArrayOf(segStart, x - 1))
|
||||
}
|
||||
inSeg = false
|
||||
}
|
||||
}
|
||||
if (inSeg) {
|
||||
if (segs.isNotEmpty() && (w - 1) - segs.last()[1] < DET_MERGE_GAP) {
|
||||
segs.last()[1] = w - 1
|
||||
} else {
|
||||
segs.add(intArrayOf(segStart, w - 1))
|
||||
}
|
||||
}
|
||||
for ((x0, x1) in segs.map { it[0] to it[1] }) {
|
||||
// 过滤过小的框
|
||||
val boxW = x1 - x0 + 1
|
||||
val boxH = y1 - y0 + 1
|
||||
if (boxW < 4 || boxH < 2) continue
|
||||
// 映射回原图坐标(4 个角点)
|
||||
val fx0 = x0 * ratioX
|
||||
val fx1 = x1 * ratioX
|
||||
val fy0 = y0 * ratioY
|
||||
val fy1 = y1 * ratioY
|
||||
boxes.add(listOf(
|
||||
floatArrayOf(fx0, fy0),
|
||||
floatArrayOf(fx1, fy0),
|
||||
floatArrayOf(fx1, fy1),
|
||||
floatArrayOf(fx0, fy1),
|
||||
))
|
||||
}
|
||||
}
|
||||
return boxes
|
||||
}
|
||||
|
||||
// ============== CTC 解码 ==============
|
||||
|
||||
/**
|
||||
* CTC greedy decode:每个时间步取 argmax,去 blank 去重复。返回 (text, avgConfidence)。
|
||||
*
|
||||
* PaddleOCR 约定:logits 的 index 0 固定是 blank,字符从 index 1 起,
|
||||
* dictionary[i] 对应模型输出 index i+1。因此 dictIdx = argmaxIdx - 1。
|
||||
* 已用 onnxruntime 实证:argmax 序列中 0 占多数(即 blank),真实字符索引
|
||||
* (如 90→'支')按 idx-1 映射到 dictionary 即可正确还原中文。
|
||||
*/
|
||||
private fun ctcGreedyDecode(logits: Array<FloatArray>): Pair<String, Float> {
|
||||
if (logits.isEmpty()) return "" to 0f
|
||||
val numClasses = logits[0].size
|
||||
val blankIdx = 0 // PaddleOCR CTC:blank 固定在 index 0
|
||||
|
||||
val sb = StringBuilder()
|
||||
var lastIdx = -1
|
||||
var confSum = 0.0f
|
||||
var confCount = 0
|
||||
for (t in logits.indices) {
|
||||
var maxIdx = 0
|
||||
var maxVal = logits[t][0]
|
||||
for (i in 1 until numClasses) {
|
||||
if (logits[t][i] > maxVal) { maxVal = logits[t][i]; maxIdx = i }
|
||||
}
|
||||
// softmax 概率(用于置信度统计)
|
||||
var expSum = 0.0
|
||||
for (i in 0 until numClasses) expSum += Math.exp(logits[t][i].toDouble())
|
||||
val prob = Math.exp(maxVal.toDouble()) / expSum
|
||||
|
||||
if (maxIdx != blankIdx && maxIdx != lastIdx) {
|
||||
val dictIdx = maxIdx - 1 // index 1..N → dictionary[0..N-1]
|
||||
if (dictIdx in 0 until dictionary.size) {
|
||||
sb.append(dictionary[dictIdx])
|
||||
confSum += prob.toFloat()
|
||||
confCount++
|
||||
}
|
||||
}
|
||||
lastIdx = maxIdx
|
||||
}
|
||||
val avgConf = if (confCount > 0) confSum / confCount else 0f
|
||||
return sb.toString() to avgConf
|
||||
}
|
||||
|
||||
// ============== Bitmap 工具 ==============
|
||||
|
||||
private fun cropBox(bmp: Bitmap, box: List<FloatArray>): Bitmap? {
|
||||
val xs = box.map { it[0] }
|
||||
val ys = box.map { it[1] }
|
||||
val paddingX = 4
|
||||
val paddingY = 2
|
||||
val minX = max(0, (xs.minOrNull() ?: 0f).toInt() - paddingX)
|
||||
val minY = max(0, (ys.minOrNull() ?: 0f).toInt() - paddingY)
|
||||
val maxX = min(bmp.width, ((xs.maxOrNull() ?: 0f) + 1).toInt() + paddingX)
|
||||
val maxY = min(bmp.height, ((ys.maxOrNull() ?: 0f) + 1).toInt() + paddingY)
|
||||
val w = maxX - minX
|
||||
val h = maxY - minY
|
||||
if (w < 2 || h < 2) return null
|
||||
return Bitmap.createBitmap(bmp, minX, minY, w, h)
|
||||
}
|
||||
|
||||
private fun resizeForDet(bmp: Bitmap, maxSide: Int): Bitmap {
|
||||
val ratio = maxSide.toFloat() / max(bmp.width, bmp.height)
|
||||
if (ratio >= 1f) return bmp
|
||||
val newW = (bmp.width * ratio).toInt()
|
||||
val newH = (bmp.height * ratio).toInt()
|
||||
// 确保尺寸是 32 的倍数(det 模型下采样要求)
|
||||
val alignedW = (newW / 32) * 32
|
||||
val alignedH = (newH / 32) * 32
|
||||
if (alignedW < 32 || alignedH < 32) return bmp
|
||||
return Bitmap.createScaledBitmap(bmp, alignedW, alignedH, true)
|
||||
}
|
||||
|
||||
// ============== 公共工具(复用) ==============
|
||||
|
||||
/** 解码 base64 图片为 Bitmap。 */
|
||||
private fun decodeBase64(base64: String): Bitmap? {
|
||||
return try {
|
||||
// 去除 data:image/...;base64, 前缀
|
||||
val data = if (base64.contains(",")) base64.substringAfter(",") else base64
|
||||
val bytes = Base64.decode(data, Base64.DEFAULT)
|
||||
BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
|
||||
} catch (e: Exception) {
|
||||
Log.e(OCR_MODULE_NAME, "base64 解码失败: ${e.message}")
|
||||
null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 短边压缩到 maxShortEdge(参考 AutoAccounting scaleDownForOcr)。
|
||||
* 像素量比 1440p 减少约 75%,识别速度大幅提升。
|
||||
*/
|
||||
private fun scaleDownForOcr(bitmap: Bitmap, maxShortEdge: Int): Bitmap {
|
||||
val width = bitmap.width
|
||||
val height = bitmap.height
|
||||
val shortEdge = minOf(width, height)
|
||||
if (shortEdge <= maxShortEdge) return bitmap
|
||||
val scale = maxShortEdge.toFloat() / shortEdge
|
||||
val newWidth = (width * scale).toInt()
|
||||
val newHeight = (height * scale).toInt()
|
||||
return Bitmap.createScaledBitmap(bitmap, newWidth, newHeight, true)
|
||||
}
|
||||
|
||||
override fun onCatalystInstanceDestroy() {
|
||||
super.onCatalystInstanceDestroy()
|
||||
release()
|
||||
}
|
||||
|
||||
override fun invalidate() {
|
||||
super.invalidate()
|
||||
release()
|
||||
}
|
||||
|
||||
private fun release() {
|
||||
lock.lock()
|
||||
try {
|
||||
detSession?.close()
|
||||
recSession?.close()
|
||||
// OrtEnvironment 是单例,不主动 close(进程级)
|
||||
detSession = null
|
||||
recSession = null
|
||||
ortEnv = null
|
||||
initialized = false
|
||||
} catch (_: Exception) {
|
||||
} finally {
|
||||
lock.unlock()
|
||||
}
|
||||
}
|
||||
|
||||
/** 识别结果块。 */
|
||||
private data class OcrBlock(val text: String, val x: Float, val y: Float, val width: Float, val height: Float, val confidence: Float)
|
||||
/** 预处理后的张量数据 + 宽高。 */
|
||||
private data class TensorData(val data: FloatArray, val w: Int, val h: Int)
|
||||
|
||||
companion object {
|
||||
/** OCR 最大短边(参考 AutoAccounting OCR_MAX_SHORT_EDGE)。 */
|
||||
private const val OCR_MAX_SHORT_EDGE = 720
|
||||
/** det resize 最大边(PaddleOCR limit_max_side_len 默认值)。 */
|
||||
private const val DET_LIMIT_MAX_SIDE = 960
|
||||
/** DB 二值化阈值。 */
|
||||
private const val DET_THRESH = 0.3f
|
||||
/** 投影法合并相邻文本段的间隔(像素)。 */
|
||||
private const val DET_MERGE_GAP = 10
|
||||
/** rec 固定图像高度。 */
|
||||
private const val REC_IMAGE_HEIGHT = 48
|
||||
/** rec 单行最大宽度。 */
|
||||
private const val REC_MAX_WIDTH = 320
|
||||
/** assets 中的模型/字典文件名。 */
|
||||
private const val ASSET_DET_MODEL = "ppocrv5_det.onnx"
|
||||
private const val ASSET_REC_MODEL = "ppocrv5_rec.onnx"
|
||||
// PP-OCRv5 多语言识别模型的配套字典(18383 字符 + 运行时 1 blank = 18385 维输出)。
|
||||
// 注意:必须与 rec 模型配套,错用旧版 ppocr_keys_v1.txt(6623)会导致 CTC 解码乱码。
|
||||
private const val ASSET_DICT = "ppocrv5_dict.txt"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package com.beancount.mobile.ppocr
|
||||
|
||||
import android.view.View
|
||||
import com.facebook.react.ReactPackage
|
||||
import com.facebook.react.bridge.NativeModule
|
||||
import com.facebook.react.bridge.ReactApplicationContext
|
||||
import com.facebook.react.uimanager.ReactShadowNode
|
||||
import com.facebook.react.uimanager.ViewManager
|
||||
|
||||
/**
|
||||
* 注册 OcrModule 到 React Native 的 Package(plan.md「3.2 Config Plugin」)。
|
||||
*
|
||||
* 由 app.plugin.js 的 withMainApplication 注入到 MainApplication.getPackages() 列表。
|
||||
* RN 在启动时遍历所有 Package,调用 createNativeModules 注册原生模块。
|
||||
*/
|
||||
class OcrPackage : ReactPackage {
|
||||
override fun createNativeModules(rc: ReactApplicationContext): List<NativeModule> {
|
||||
return listOf(OcrModule(rc))
|
||||
}
|
||||
|
||||
override fun createViewManagers(rc: ReactApplicationContext): List<ViewManager<View, ReactShadowNode<*>>> {
|
||||
return emptyList()
|
||||
}
|
||||
}
|
||||
@@ -103,11 +103,10 @@ function withPpOcr(config) {
|
||||
config = withAppBuildGradle(config, (modConfig) => {
|
||||
let gradle = modConfig.modResults.contents;
|
||||
if (!gradle.includes('onnxruntime')) {
|
||||
// 在 dependencies { ... } 块末尾追加
|
||||
// 在 dependencies { 开头处插入,避免嵌套花括号的正则匹配错误
|
||||
gradle = gradle.replace(
|
||||
/dependencies\s*{([\s\S]*?)^\s*}/m,
|
||||
(m, inner) =>
|
||||
`dependencies {${inner}\n // PP-OCRv5 ONNX Runtime(由 Config Plugin 注入)\n implementation 'com.microsoft.onnxruntime:onnxruntime-android:1.20.0'\n}`,
|
||||
/(dependencies\s*\{)/,
|
||||
`$1\n // PP-OCRv5 ONNX Runtime(由 Config Plugin 注入)\n implementation 'com.microsoft.onnxruntime:onnxruntime-android:1.20.0'`
|
||||
);
|
||||
}
|
||||
modConfig.modResults.contents = gradle;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user