批量OCR期间高峰内存
同时运行多个PDF段段段段的OCR会增加内存使用。 每个任务通过IronTesseract引擎重新加载语言模型文件。在处理器满负荷并发时,这会将峰值内存推入多GB范围,导致在内存有限的环境中失败。
OCR本身是内存密集型的。 每个IronTesseract引擎加载语言模型文件到内存中。 为每个段段段段创建一个新引擎会重复加载那些模型,并在每个CPU核心运行一个OCR任务(Environment.ProcessorCount),让许多位图密集型的作业并行运行。
修复方法是限制正在进行的作业数量:限制并发,通过池重用引擎,并用信号量闸口工作。
解决方案
1. 限制OCR并发数
限制同时运行的OCR任务数量到一个小值。 较少的并发任务意味着内存中同时只有较少的整页位图,直接降低峰值。 根据机器的能力调整上限。
// Clamp concurrency to avoid memory saturation and CPU over-subscription.
int concurrency = Math.Clamp(Environment.ProcessorCount / 2, 1, 4);
// Clamp concurrency to avoid memory saturation and CPU over-subscription.
int concurrency = Math.Clamp(Environment.ProcessorCount / 2, 1, 4);
Imports System
' Clamp concurrency to avoid memory saturation and CPU over-subscription.
Dim concurrency As Integer = Math.Clamp(Environment.ProcessorCount \ 2, 1, 4)
2. 池化引擎
在启动时为每个并发插槽创建一个IronTesseract引擎,并跨每个段段段段重用它们,而不是每次构建一个新引擎并重新加载语言模型。
// Pre-create one engine per concurrent slot and reuse them across segments.
var enginePool = new ConcurrentBag<IronTesseract>(
Enumerable.Range(0, concurrency).Select(_ => new IronTesseract())
);
// Pre-create one engine per concurrent slot and reuse them across segments.
var enginePool = new ConcurrentBag<IronTesseract>(
Enumerable.Range(0, concurrency).Select(_ => new IronTesseract())
);
' Pre-create one engine per concurrent slot and reuse them across segments.
Dim enginePool As New ConcurrentBag(Of IronTesseract)(
Enumerable.Range(0, concurrency).Select(Function(_) New IronTesseract())
)
一次性构建池会将语言模型加载成本摊在整个运行中,而不是每段段段段都支付。
3. 使用信号量控制作业
初始化一个设置为并发限制的using中。 每个任务在开始时调用Release(),因此只有允许的段段段段数量在运行中。
using var semaphore = new SemaphoreSlim(concurrency);
await semaphore.WaitAsync();
try
{
// Rent a pre-loaded engine from the pool.
if (!enginePool.TryTake(out var ocr))
ocr = new IronTesseract(); // Defensive fallback; should never be reached.
try
{
using var input = new OcrInput();
input.LoadPdf(segmentStream); // page-range segment produced upstream
var ocrResult = await ocr.ReadAsync(input);
ocrResult.SaveAsSearchablePdf(outputPath);
}
finally
{
enginePool.Add(ocr); // Return engine to pool for the next waiting segment.
}
}
finally
{
semaphore.Release();
}
using var semaphore = new SemaphoreSlim(concurrency);
await semaphore.WaitAsync();
try
{
// Rent a pre-loaded engine from the pool.
if (!enginePool.TryTake(out var ocr))
ocr = new IronTesseract(); // Defensive fallback; should never be reached.
try
{
using var input = new OcrInput();
input.LoadPdf(segmentStream); // page-range segment produced upstream
var ocrResult = await ocr.ReadAsync(input);
ocrResult.SaveAsSearchablePdf(outputPath);
}
finally
{
enginePool.Add(ocr); // Return engine to pool for the next waiting segment.
}
}
finally
{
semaphore.Release();
}
Imports System.Threading
Imports IronOcr
Dim semaphore As New SemaphoreSlim(concurrency)
Await semaphore.WaitAsync()
Try
' Rent a pre-loaded engine from the pool.
Dim ocr As IronTesseract = Nothing
If Not enginePool.TryTake(ocr) Then
ocr = New IronTesseract() ' Defensive fallback; should never be reached.
End If
Try
Using input As New OcrInput()
input.LoadPdf(segmentStream) ' page-range segment produced upstream
Dim ocrResult = Await ocr.ReadAsync(input)
ocrResult.SaveAsSearchablePdf(outputPath)
End Using
Finally
enginePool.Add(ocr) ' Return engine to pool for the next waiting segment.
End Try
Finally
semaphore.Release()
End Try
finally把预加载的引擎直接交给下一个等待的段段段段。
4. 为每个段段段段处置OcrInput
将每个using中,这样在读取段段段段时会立即释放被渲染的页面位图,在下一个任务索取插槽前。
OcrInput是阻止位图内存在段段段段之间积累的关键点; 没有它,释放的插槽仍然保留其页面位图。

