LemonWiki共筆 - Recent changes [en]

排除 MySQL 技術疑難問題

Planetoid — Fri, 19 Jun 2026 15:28:31 GMT

錯誤代碼: 13 拒絕存取

← Older revision		Revision as of 23:28, 19 June 2026
Line 566:		Line 566:
	$ sudo chmod 774 /Applications/XAMPP/xamppfiles/var/mysql/XXXMacBook-Pro.local.err		$ sudo chmod 774 /Applications/XAMPP/xamppfiles/var/mysql/XXXMacBook-Pro.local.err
	</pre>		</pre>

			=== 錯誤代碼: 11 資源暫時無法取得 ===
			錯誤訊息：'''Can't create a new thread (errno 11 "Resource temporarily unavailable")'''

			解決方案：[https://errerrors.blogspot.com/2026/06/xampp-mariadb-errno-11-open-files-limit.html 解決 XAMPP MariaDB 遇到 errno 11「Resource temporarily unavailable」錯誤]

	=== 錯誤代碼: 13 拒絕存取 ===		=== 錯誤代碼: 13 拒絕存取 ===
Line 597:		Line 602:

	以管理員特權開啟命令行。（如何操作: [https://www.howtogeek.com/194041/how-to-open-the-command-prompt-as-administrator-in-windows-8.1/ 如何在 Windows 8 或 10 中以管理員身份開啟命令提示符]）		以管理員特權開啟命令行。（如何操作: [https://www.howtogeek.com/194041/how-to-open-the-command-prompt-as-administrator-in-windows-8.1/ 如何在 Windows 8 或 10 中以管理員身份開啟命令提示符]）


	=== 錯誤！：SQLSTATE[HY000]：一般錯誤：3 寫入檔案 'xxx\Temp\xxx.tmp' 時出錯（錯誤代碼：28 - 設備上沒有剩餘空間） ===		=== 錯誤！：SQLSTATE[HY000]：一般錯誤：3 寫入檔案 'xxx\Temp\xxx.tmp' 時出錯（錯誤代碼：28 - 設備上沒有剩餘空間） ===

Troubleshooting of MySQL errors

Planetoid — Fri, 19 Jun 2026 15:26:39 GMT

errno 41 - Error dropping database

← Older revision		Revision as of 23:26, 19 June 2026
Line 649:		Line 649:
	Validation		Validation
	* Query the syntax {{kbd \| key=<nowiki>SHOW VARIABLES LIKE 'tmpdir';</nowiki>}} to validate the modification of mysql configuration file.		* Query the syntax {{kbd \| key=<nowiki>SHOW VARIABLES LIKE 'tmpdir';</nowiki>}} to validate the modification of mysql configuration file.

			=== errno 11 - Resource temporarily unavailable ===
			Message: '''Can't create a new thread (errno 11 "Resource temporarily unavailable")'''

			Solution: [https://duntuk.com/how-raise-ulimit-open-files-and-mysql-openfileslimit How to permanently raise ulimit 'open files' and MySQL 'open_files_limit']

	=== errno 41 - Error dropping database ===		=== errno 41 - Error dropping database ===

User:Planetoid

Planetoid — Thu, 18 Jun 2026 14:38:46 GMT

social

← Older revision		Revision as of 22:38, 18 June 2026
Line 41:		Line 41:
	* [http://zh.scratchpad.wikia.com/wiki/Special:Contributions/Planetoid my contribution][http://scratchpad.wikia.com/wiki/Special:Contributions/Planetoid] to [http://zh.scratchpad.wikia.com/wiki/%E5%9C%8D%E7%B4%80%E5%AF%A6%E9%A9%97%E5%AE%A4 圍紀實驗室]/[http://scratchpad.wikia.com/wiki/Scratchpad_Wiki_Labs Scratchpad]		* [http://zh.scratchpad.wikia.com/wiki/Special:Contributions/Planetoid my contribution][http://scratchpad.wikia.com/wiki/Special:Contributions/Planetoid] to [http://zh.scratchpad.wikia.com/wiki/%E5%9C%8D%E7%B4%80%E5%AF%A6%E9%A9%97%E5%AE%A4 圍紀實驗室]/[http://scratchpad.wikia.com/wiki/Scratchpad_Wiki_Labs Scratchpad]
	* [http://userstyles.org/users/3650 planetoid - userstyles.org]		* [http://userstyles.org/users/3650 planetoid - userstyles.org]
	* [https://greasyfork.org/zh-TW/users/160335-planetoid greasyfork] ~~(user scripts)~~		* user scripts hosted by [https://greasyfork.org/zh-TW/users/160335-planetoid greasyfork]
	** [https://greasyfork.org/zh-TW/scripts/506037-hackmd-copy-code-snippet HackMD Copy Code Snippet] MIT License		** [https://greasyfork.org/zh-TW/scripts/506037-hackmd-copy-code-snippet HackMD Copy Code Snippet] MIT License
	** [https://greasyfork.org/zh-TW/scripts/570350-hide-referrer Hide Referrer] MIT License		** [https://greasyfork.org/zh-TW/scripts/570350-hide-referrer Hide Referrer] MIT License

Troubleshooting of GIT

Planetoid — Wed, 17 Jun 2026 08:04:55 GMT

SSH Authentication Failed: Permission denied (publickey)

← Older revision		Revision as of 16:04, 17 June 2026
Line 60:		Line 60:

	{{exclaim}} Notice: `-o IdentitiesOnly=yes` prevents SSH from trying other keys loaded in the agent, which can cause unexpected auth failures.		{{exclaim}} Notice: `-o IdentitiesOnly=yes` prevents SSH from trying other keys loaded in the agent, which can cause unexpected auth failures.


			=== Git Submodule Pull Failed: unable to create file — File exists ===

			'''Error Message'''

			After pulling in GitHub Desktop, met the error message

			<pre>
			Fetching submodule lib
			error: unable to create file feeds/example.php: File exists
			Updating xxxxxxxx..xxxxxxxx
			</pre>

			'''Cause'''

			When Git tries to update the submodule to a newer commit, it finds a local untracked file with the same name, blocking the checkout. A mid-pull failure leaves the submodule in an inconsistent state and causes a large number of unexpected unstaged changes to appear in GitHub Desktop.

			'''Solution'''

			Run the following command in the project root directory to force re-sync the submodule:

			<pre>
			git submodule update --init --recursive --force
			</pre>

			If it still fails, manually remove the conflicting file and retry:

			<pre>
			rm lib/feeds/example.php
			git submodule update --init --recursive
			</pre>

			Enter the submodule directory and verify the status is clean:

			<pre>
			cd lib
			git status
			</pre>

			The expected output should be:

			<pre>
			HEAD detached from xxxxxxxx
			nothing to commit, working tree clean
			</pre>

			`nothing to commit, working tree clean` confirms the submodule has been successfully synced — return to GitHub Desktop and continue as normal. `HEAD detached` is the expected state for a submodule and does not require any action.

			{{exclaim}} Notice: The large number of unstaged changes appearing in GitHub Desktop after a failed Pull will typically return to normal once the submodule is fixed. If unexpected changes still remain, run <code>git submodule foreach git checkout .</code> to revert all modifications inside every submodule.

	== Further Reading ==		== Further Reading ==

Troubleshooting of PostgreSQL

Planetoid — Tue, 16 Jun 2026 07:17:54 GMT

pgAdmin 4 on macOS Has No Autocomplete for Field or Table Names

← Older revision		Revision as of 15:17, 16 June 2026
(2 intermediate revisions by the same user not shown)
Line 1:		Line 1:
	Troubleshooting of [https://www.postgresql.org/ PostgreSQL]		Troubleshooting of [https://www.postgresql.org/ PostgreSQL]

	== ~~How to fix Syntax error~~ of PostgreSQL ==		== Troubleshooting of PostgreSQL ==
	=== How to fix Syntax error: 7 ERROR: LIMIT #,# syntax is not supported ===		=== How to fix Syntax error: 7 ERROR: LIMIT #,# syntax is not supported ===
	Query syntax mer error		Query syntax mer error
Line 87:		Line 87:
	=== How to fix: The numeric column data all became null after imported ===		=== How to fix: The numeric column data all became null after imported ===
	[[Troubleshooting of Navicat for PostgreSQL]]		[[Troubleshooting of Navicat for PostgreSQL]]


			=== pgAdmin 4 on macOS Has No Autocomplete for Field or Table Names ===

			[https://www.pgadmin.org/download/ pgAdmin] 4 has autocomplete built in by default, but it sometimes needs to be triggered manually or configured first.

			'''How to trigger it'''
			In the Query Tool, press `Ctrl / Cmd + Space`. However, on macOS, `Ctrl + Space` is reserved for switching input methods, and `Cmd + Space` opens Spotlight, so the shortcut conflict likely prevents the autocomplete menu from appearing. This menu is what normally shows suggestions for tables, columns, schemas, and keywords.

			'''How to make autocomplete appear as you type'''

			# Windows / Linux: Go to File → Preferences → Query Tool → Auto Completion, and make sure "Keywords autocomplete?" is checked.
			# Mac: Go to pgAdmin 4 → Preferences → Query Tool → Auto Completion, and make sure "Keywords autocomplete?" is checked.

	== Further reading ==		== Further reading ==

PDF Table Parsing

Planetoid — Mon, 15 Jun 2026 12:48:04 GMT

1. Terminology

← Older revision		Revision as of 20:48, 15 June 2026
(2 intermediate revisions by the same user not shown)
Line 1:		Line 1:
	== Technical Notes on PDF Table Parsing ==		Technical Notes on PDF Table Parsing

	PDF table parsing usually cannot rely solely on the raw output produced by tools such as <code>pdfplumber.extract_tables()</code>, Camelot, Tabula, or similar libraries. Since PDF is primarily a layout-oriented format rather than a structured data format, practical implementations often require additional rules, state management, and post-processing steps in order to produce stable and usable datasets.		PDF table parsing usually cannot rely solely on the raw output produced by tools such as <code>pdfplumber.extract_tables()</code>, Camelot, Tabula, or similar libraries. Since PDF is primarily a layout-oriented format rather than a structured data format, practical implementations often require additional rules, state management, and post-processing steps in order to produce stable and usable datasets.

	~~The following notes summarize common technical issues that can serve as a reference when developing a PDF table parser.~~		=== Comparison of Different Parsers ===

	# ~~Terminology~~		# [https://github.com/jsvine/pdfplumber pdfplumber] requires writing code to process the data.
	# ~~Section Detection Requires Cross~~-~~Page State~~		# [https://www.xpdfreader.com/pdftotext-man.html pdftotext] with the `-layout` option: The drawback is that columns are aligned using spaces, rather than being extracted as a truly structured table.
	~~# A Single Section May Contain Multiple Table Schemas~~		# [https://jina.ai/ jina.ai parser]: The drawback is that columns are aligned using spaces, rather than being extracted as a truly structured table.
	~~# Parent~~-~~Child Row Relationships Require Forward-Fill Support~~
	# ~~Cross-Page Continuation Is More Than Repeated Headers~~
	~~# Headers, Notes~~, ~~and Data Rows Should Be Detected in Stages~~
	~~# Metadata Should Be Stored at the Record Level~~
	~~# Implementation Recommendations~~

	=== 1. Terminology ===		=== 1. Terminology ===

	~~====~~ Parser ~~====~~		'''Parser'''

	A <code>parser</code> is a program responsible for parsing PDF content and converting it into structured data. It usually does more than simply read text or tables; it also needs to identify sections, table types, column positions, data rows, continuations, and source metadata.		A <code>parser</code> is a program responsible for parsing PDF content and converting it into structured data. It usually does more than simply read text or tables; it also needs to identify sections, table types, column positions, data rows, continuations, and source metadata.

	~~====~~ Record ~~====~~		'''Record'''

	A <code>record</code> is a standardized data entry produced by the parser.		A <code>record</code> is a standardized data entry produced by the parser.
Line 26:		Line 21:
	A single row in a PDF does not necessarily correspond to one record. The same record may be split across multiple rows, continue across pages, or require inherited information from a parent row before it becomes complete.		A single row in a PDF does not necessarily correspond to one record. The same record may be split across multiple rows, continue across pages, or require inherited information from a parent row before it becomes complete.

	~~====~~ Section / Subsection ~~====~~		'''Section / Subsection'''

	<code>section</code> and <code>subsection</code> refer to the section and subsection in a document.		<code>section</code> and <code>subsection</code> refer to the section and subsection in a document.
Line 32:		Line 27:
	These fields are typically used to describe the source context of the data, such as which section or table block a particular record comes from.		These fields are typically used to describe the source context of the data, such as which section or table block a particular record comes from.

	~~====~~ Table Type ~~====~~		'''Table Type'''

	<code>table_type</code> is a table-type label defined by the parser to indicate what kind of data structure the current table represents.		<code>table_type</code> is a table-type label defined by the parser to indicate what kind of data structure the current table represents.
Line 48:		Line 43:
	It is not a built-in PDF field, nor is it information automatically provided by <code>pdfplumber</code>, Camelot, or Tabula.		It is not a built-in PDF field, nor is it information automatically provided by <code>pdfplumber</code>, Camelot, or Tabula.

	~~====~~ Table Schema ~~====~~		'''Table Schema'''

	A <code>table schema</code> refers to the column structure and parsing rules for a specific type of table.		A <code>table schema</code> refers to the column structure and parsing rules for a specific type of table.
Line 56:		Line 51:
	In PDF table parsing, a <code>table schema</code> is not necessarily a database schema. Instead, it is a set of rules used by the parser to align and standardize table data.		In PDF table parsing, a <code>table schema</code> is not necessarily a database schema. Instead, it is a set of rules used by the parser to align and standardize table data.

	~~====~~ Column Mapping ~~====~~		'''Column Mapping'''

	<code>column mapping</code> refers to the process of mapping original PDF columns to standardized fields.		<code>column mapping</code> refers to the process of mapping original PDF columns to standardized fields.
Line 62:		Line 57:
	For example, different documents may use headers such as “Item Name,” “Object Name,” or “Target Name,” but all of them can be normalized into <code>item_name</code> in the output data.		For example, different documents may use headers such as “Item Name,” “Object Name,” or “Target Name,” but all of them can be normalized into <code>item_name</code> in the output data.

	~~====~~ Alias Mapping ~~====~~		'''Alias Mapping'''

	<code>alias mapping</code> refers to a lookup table for header aliases.		<code>alias mapping</code> refers to a lookup table for header aliases.
Line 68:		Line 63:
	Because PDF table headers often vary across versions, formats, or reporting units, the parser needs to map multiple header names to the same standardized field.		Because PDF table headers often vary across versions, formats, or reporting units, the parser needs to map multiple header names to the same standardized field.

	~~====~~ Forward-Fill ~~====~~		'''Forward-Fill'''

	<code>forward-fill</code> refers to the process of allowing later rows to inherit field values from previous rows.		<code>forward-fill</code> refers to the process of allowing later rows to inherit field values from previous rows.
Line 74:		Line 69:
	In a parent-child row structure, a parent row may appear only once, while subsequent child rows omit repeated information. In this case, the parser needs to fill the parent-row information into the child rows.		In a parent-child row structure, a parent row may appear only once, while subsequent child rows omit repeated information. In this case, the parser needs to fill the parent-row information into the child rows.

	~~====~~ Parent Row / Child Row ~~====~~		'''Parent Row / Child Row'''

	A <code>parent row</code> is a row that provides main category, group, or summary information.		A <code>parent row</code> is a row that provides main category, group, or summary information.
Line 82:		Line 77:
	In PDF tables, it is common for a parent row to list primary information while child rows only list detailed items. Without handling parent-child row relationships, the output data may lose necessary context.		In PDF tables, it is common for a parent row to list primary information while child rows only list detailed items. Without handling parent-child row relationships, the output data may lose necessary context.

	~~====~~ Cross-Page Continuation ~~====~~		'''Cross-Page Continuation'''

	<code>cross-page continuation</code> refers to a situation where the same record is split and continues onto the next page.		<code>cross-page continuation</code> refers to a situation where the same record is split and continues onto the next page.
Line 90:		Line 85:
	Cross-page continuation handling requires determining whether text on the next page belongs to an unfinished record from the previous page, instead of treating it directly as a new data row.		Cross-page continuation handling requires determining whether text on the next page belongs to an unfinished record from the previous page, instead of treating it directly as a new data row.

	~~====~~ Metadata ~~====~~		'''Metadata'''

	<code>metadata</code> refers to auxiliary information that describes the data source and parsing state.		<code>metadata</code> refers to auxiliary information that describes the data source and parsing state.
Line 219:		Line 214:

	[[Category: Tool]]		[[Category: Tool]]
			[[Category: Revised with LLMs]]

PDF Table Parsing

Planetoid — Mon, 15 Jun 2026 10:31:38 GMT

Created page with "== Technical Notes on PDF Table Parsing == PDF table parsing usually cannot rely solely on the raw output produced by tools such as <code>pdfplumber.extract_tables()</code>, Camelot, Tabula, or similar libraries. Since PDF is primarily a layout-oriented format rather than a structured data format, practical implementations often require additional rules, state management, and post-processing steps in order to produce stable and usable datasets. The following notes summ..."

New page

== Technical Notes on PDF Table Parsing ==

PDF table parsing usually cannot rely solely on the raw output produced by tools such as <code>pdfplumber.extract_tables()</code>, Camelot, Tabula, or similar libraries. Since PDF is primarily a layout-oriented format rather than a structured data format, practical implementations often require additional rules, state management, and post-processing steps in order to produce stable and usable datasets.

The following notes summarize common technical issues that can serve as a reference when developing a PDF table parser.

# Terminology
# Section Detection Requires Cross-Page State
# A Single Section May Contain Multiple Table Schemas
# Parent-Child Row Relationships Require Forward-Fill Support
# Cross-Page Continuation Is More Than Repeated Headers
# Headers, Notes, and Data Rows Should Be Detected in Stages
# Metadata Should Be Stored at the Record Level
# Implementation Recommendations

=== 1. Terminology ===

==== Parser ====

A <code>parser</code> is a program responsible for parsing PDF content and converting it into structured data. It usually does more than simply read text or tables; it also needs to identify sections, table types, column positions, data rows, continuations, and source metadata.

==== Record ====

A <code>record</code> is a standardized data entry produced by the parser.

A single row in a PDF does not necessarily correspond to one record. The same record may be split across multiple rows, continue across pages, or require inherited information from a parent row before it becomes complete.

==== Section / Subsection ====

<code>section</code> and <code>subsection</code> refer to the section and subsection in a document.

These fields are typically used to describe the source context of the data, such as which section or table block a particular record comes from.

==== Table Type ====

<code>table_type</code> is a table-type label defined by the parser to indicate what kind of data structure the current table represents.

Examples include:

* <code>summary_table</code>
* <code>detail_table</code>
* <code>parent_child_table</code>
* <code>cross_reference_table</code>
* <code>unknown_table</code>

The purpose of <code>table_type</code> is to help the parser determine which set of parsing rules should be applied.

It is not a built-in PDF field, nor is it information automatically provided by <code>pdfplumber</code>, Camelot, or Tabula.

==== Table Schema ====

A <code>table schema</code> refers to the column structure and parsing rules for a specific type of table.

It usually includes the number of columns, column order, standardized column names, column aliases, required fields, nullable fields, and whether special continuation handling or parent-child row handling is needed.

In PDF table parsing, a <code>table schema</code> is not necessarily a database schema. Instead, it is a set of rules used by the parser to align and standardize table data.

==== Column Mapping ====

<code>column mapping</code> refers to the process of mapping original PDF columns to standardized fields.

For example, different documents may use headers such as “Item Name,” “Object Name,” or “Target Name,” but all of them can be normalized into <code>item_name</code> in the output data.

==== Alias Mapping ====

<code>alias mapping</code> refers to a lookup table for header aliases.

Because PDF table headers often vary across versions, formats, or reporting units, the parser needs to map multiple header names to the same standardized field.

==== Forward-Fill ====

<code>forward-fill</code> refers to the process of allowing later rows to inherit field values from previous rows.

In a parent-child row structure, a parent row may appear only once, while subsequent child rows omit repeated information. In this case, the parser needs to fill the parent-row information into the child rows.

==== Parent Row / Child Row ====

A <code>parent row</code> is a row that provides main category, group, or summary information.

A <code>child row</code> is a detail row that belongs to a parent row.

In PDF tables, it is common for a parent row to list primary information while child rows only list detailed items. Without handling parent-child row relationships, the output data may lose necessary context.

==== Cross-Page Continuation ====

<code>cross-page continuation</code> refers to a situation where the same record is split and continues onto the next page.

This may occur in names, descriptive text, category fields, remarks, or other long fields.

Cross-page continuation handling requires determining whether text on the next page belongs to an unfinished record from the previous page, instead of treating it directly as a new data row.

==== Metadata ====

<code>metadata</code> refers to auxiliary information that describes the data source and parsing state.

Common fields include <code>page</code>, <code>table_index</code>, <code>table_type</code>, <code>section</code>, <code>subsection</code>, <code>source_file</code>, <code>raw_row</code>, and <code>parse_warning</code>.

Metadata helps with debugging, auditing, data traceability, and data quality checks.

=== 2. Section Detection Requires Cross-Page State ===

Section, subsection, and table category information in PDF documents may not be repeated on every page. Therefore, the parser needs to preserve the current <code>section</code>, <code>subsection</code>, or other document-level context and reuse it on subsequent pages.

Common scenarios include:

* Continuation pages may contain only table content without repeating the section title.
* Table of contents, summary, or explanatory pages may contain multiple section names at once. If the parser only checks whether a page contains certain keywords, it may misclassify the page.
* Section titles may use different punctuation or spacing due to layout variations, such as full-width or half-width parentheses, enumeration marks, colons, full-width periods, or extra spaces.
* When switching sections, subsections, or table types, inherited state from the previous table must be cleared to prevent data from being incorrectly carried over into the next block.

A recommended approach is to determine sections using a combination of text position, title patterns, and contextual state, rather than relying only on keyword matching across the entire page.

=== 3. A Single Section May Contain Multiple Table Schemas ===

A single section may contain multiple types of tables. If the parser determines column mapping only from the section name, columns may be misaligned.

When parsing each table, the parser should first inspect the header or the first few rows of the table to determine:

* <code>table_type</code>
* Number of columns
* Column order
* Standardized column names
* Whether parent-child row relationships exist
* Whether special continuation handling is required

Header names may also contain aliases, abbreviations, or terms that vary across document versions. For example, “Item Name,” “Object Name,” and “Target Name” may refer to similar fields in different documents. The implementation should therefore maintain an alias mapping that normalizes different header names into consistent fields.

=== 4. Parent-Child Row Relationships Require Forward-Fill Support ===

Some PDF tables use parent rows to list primary information, while subsequent child rows only list detail fields. For example, a parent row may contain a main category, group name, case number, or summary information, while child rows only list sub-item numbers, detail names, or detailed content.

These tables require forward-fill support, where child rows inherit field values from the parent row.

Design considerations include:

* Parent row state should be isolated by context such as <code>section</code>, <code>subsection</code>, and <code>table_type</code>.
* Parent row state may need to persist across pages.
* A single text field should not be used as the parent row identifier, because different parent rows may have identical or similar names.
* Backfilling should consider at least the section, subsection, table type, and parent-row identifier.

When the section or table changes, parent row state should be explicitly reset to avoid carrying data from the previous table into the next one.

=== 5. Cross-Page Continuation Is More Than Repeated Headers ===

When PDF tables span multiple pages, data may be split in various ways. A parser that only handles repeated headers on the next page is usually insufficient for real-world documents.

Common cross-page continuation cases include:

* The first row on the next page contains only the latter part of a field from the previous page.
* Names, categories, descriptive text, or remarks are split across pages.
* The next page starts with multiple repeated header rows before continuing an unfinished record from the previous page.
* Continuation text and a new data-row identifier appear in the same row.
* Empty columns are compressed, causing column positions to shift.
* Table borders, line breaks, or OCR results split what was originally one row into multiple rows.

Therefore, cross-page processing should preserve original column positions and should not remove empty columns too early. When necessary, the parser should use column coordinates, column indexes, identifier patterns, and the previous record state to determine which field a continuation fragment belongs to.

Continuation merging should also depend on the type of text:

* Chinese text fragments can usually be concatenated directly.
* English text fragments usually require an inserted space.
* Multi-value fields, category fields, or remarks may need to preserve line breaks, enumeration marks, or delimiters.
* Numeric fields should avoid accidentally concatenating text fragments.

=== 6. Headers, Notes, and Data Rows Should Be Detected in Stages ===

PDF tables often contain headers, unit descriptions, notes, subtotals, totals, footers, or blank rows. If a parser treats any row containing text as a data row, it can easily produce incorrect records.

A recommended parsing flow is to separate the process into stages:

# Wait for header keywords to appear.
# Detect the header area and column names.
# Determine where data rows begin.
# Process regular data rows.
# Process cross-page continuations.
# Exclude notes, units, subtotals, totals, and footers.
# Output standardized records.

Missing fields may be filled with <code>null</code>. However, if columns have shifted, the parser must not blindly apply the schema based on the order of a compressed array, as this can misalign the entire row.

=== 7. Metadata Should Be Stored at the Record Level ===

When parsing PDF tables, source context should be preserved in addition to the extracted data itself. This metadata is important for debugging, auditing, data traceability, and downstream flattened outputs.

Each record should preferably include at least:

* <code>page</code>
* <code>table_index</code>
* <code>table_type</code>
* <code>section</code>
* <code>subsection</code>
* <code>source_file</code>
* <code>raw_row</code> or original row data
* <code>parse_warning</code> or parsing warning

Writing metadata at the record level helps prevent context loss when JSON is flattened into CSV, imported into a database, or passed around as individual records.

If the document itself contains a year, quarter, period, version, batch, or category information, that information should also be explicitly preserved to avoid mixing data from different batches.

=== 8. Implementation Recommendations ===

A PDF table parser can use a layered design that separates extraction, classification, cleaning, state management, normalization, and validation.

A recommended workflow is:

# Extract text, tables, and coordinate information from the PDF.
# Detect page sections and table blocks.
# Determine <code>table_type</code> based on header keywords.
# Apply the corresponding schema and header alias mapping.
# Identify data rows, parent rows, child rows, and continuation rows.
# Manage cross-page state, parent row state, and other document-level state.
# Convert the data into standardized records.
# Add source metadata.
# Run regression checks and data quality checks.
# Output JSON, CSV, or database-ready records.

This design reduces the risk of coupling individual rules too tightly to a specific PDF layout and makes it easier to adjust only the affected parts when the document format changes.

[[Category: Tool]]