pydiffx.utils.text¶

Utilities for processing text.

Module Attributes

`NEWLINE_FORMATS`	A mapping of newline format types to character sequences.
`BOMS`	A mapping of encodings to possible BOM markers.

Functions

`get_newline_for_type`(line_endings[, encoding])	Return the newline for a given type of line endings.
`guess_line_endings`(text[, encoding])	Return the line endings that appear to be used for text.
`split_lines`(data, newline[, keep_ends])	Split data along newline boundaries.
`strip_bom`(data, encoding)	Strip a BOM from the beginning of a string.

pydiffx.utils.text.NEWLINE_FORMATS = {'dos': '\r\n', 'unix': '\n'}¶

A mapping of newline format types to character sequences.

This contains only formats that are allowed in the line_endings= option in DiffX content sections.

Type:: dict

pydiffx.utils.text.BOMS = {'utf-16': (b'\xfe\xff', b'\xff\xfe'), 'utf-16-be': (b'\xfe\xff',), 'utf-16-le': (b'\xff\xfe',), 'utf-32': (b'\x00\x00\xfe\xff', b'\xff\xfe\x00\x00'), 'utf-32-be': (b'\x00\x00\xfe\xff',), 'utf-32-le': (b'\xff\xfe\x00\x00',), 'utf-8': (b'\xef\xbb\xbf',)}¶: A mapping of encodings to possible BOM markers.

pydiffx.utils.text.split_lines(data, newline, keep_ends=False)¶

Split data along newline boundaries.

This differs from str.splitlines() in that it will split across a specific newline boundary, rather than against any sequence of newline characters.

Parameters:

data (bytes) – The data to split.
newline (bytes) – The newline character(s) used to split the data into lines.
keep_ends (bool, optional) – Whether to keep the line endings in the resulting lines.

Returns:

The split list of lines.

Return type:

list of bytes

pydiffx.utils.text.get_newline_for_type(line_endings, encoding=None)¶

Return the newline for a given type of line endings.

The resulting newline characters will be encoded into the given encoding, if specified, or as plain ASCII if not specified.

If a BOM is present in the result, it will be stripped.

Parameters:

line_endings (unicode) – The type of line endings. This will be of of LineEndings.DOS or LineEndings.UNIX.
encoding (unicode, optional) – The encoding to use for the resulting newline. If None, “ascii” will be used.

Returns:

The resulting encoded newline characters.

Return type:

bytes

Raises:

LookupError – encoding was not a valid encoding type.
ValueError – line_endings was not a valid type of line endings.

pydiffx.utils.text.guess_line_endings(text, encoding=None)¶

Return the line endings that appear to be used for text.

This will check the first line of content and see if it appears to be DOS or UNIX line endings.

If there are no newlines, UNIX line endings are assumed.

Parameters:

text (bytes or unicode) – The text to guess line endings from.
encoding (unicode, optional) – The encoding of the text, if it’s a byte string.

Returns:

A 2-tuple of:

The guessed line endings type (as a line_endings= option value).
The line ending characters (in the same string type as text).

Return type:

tuple

pydiffx.utils.text.strip_bom(data, encoding)¶

Strip a BOM from the beginning of a string.

If the encoding is one that contains a BOM, and any version (such as Big Endian or Little Endian) of the BOM are present, they’ll be stripped.

Parameters:

data (bytes) – The byte string to strip a BOM from.
encoding (unicode) – The encoding of the byte string.

Returns:

The string, without any BOM markers.

Return type:

bytes