Hyperscan basics

---requirements---

On writing this document, we're using 5.4.0 and it requires:

• CMake 2.8.11
• Ragel 6.9
• Python 2.7
• Boost 1.57

If the Hyperscan library is used on x86 systems without SSSE3, the runtime API functions will resolve to functions that return HS_ARCH_ERROR instead of potentially executing illegal instructions.

To build an AVX512VBMI runtime, the CMake variable BUILD_AVX512VBMI must be enabled manually during configuration(-DBUILD_AVX512VBMI=on).

---concepts---

--scan interface--

Patterns are provided to a compilation interface which generates an immutable pattern database. The scan interface then can be used to scan a target data buffer

--Vectored mode--

the target data consists of a list of non-contiguous blocks that are available all at once. As for block mode, no retention of state is required(Streaming mode requires retention of states).

--Stream state--

some state space is required to store data that persists between scan calls for each stream. This allows Hyperscan to track matches that span multiple blocks of data.

--All matches--

scanning /foo.*bar/ against fooxyzbarbar will return two matches from Hyperscan. at the points corresponding to the ends of fooxyzbar and fooxyzbarbar. In contrast, libpcre semantics by default would report only one match at fooxyzbarbar (greedy semantics) or, if non-greedy semantics were switched on, one match at fooxyzbar. This means that switching between greedy and non-greedy semantics is a no-op in Hyperscan.

---hs_common.h---

• HS_INVALID: A parameter passed to this function was invalid.
• hs_error_t hs_stream_size(const hs_database_t *database, size_t *stream_size): Provides the size of the stream state allocated by a single stream opened against the given database; note database decide state space's size(refer to --Stream state-- for the concept of Stream state)
• hs_error_t hs_serialized_database_size(const char *bytes, const size_t length, size_t *deserialized_size):This API can be used to allocate a (shared) memory region prior to deserializing with the hs_deserialize_database_at() function.
• hs_error_t hs_valid_platform(void): This function can be called on any x86 platform to determine if the system provides the required instruction set.

---hs_compile.h---

• HS_FLAG_MULTILINE:This flag instructs the expression to make the ^ and $ tokens match newline characters as well as the start and end of the stream.
• HS_FLAG_SINGLEMATCH: If a group of expressions sharing a match ID specify the flag, then at most one match with the match ID will be generated per stream. so better don't share ID between expressions
• HS_FLAG_COMBINATION:This flag instructs Hyperscan to parse this expression as logical combination syntax.
  

  To illustrate, here is an example combination expression: ((301 OR 302) AND 303) AND (304 OR NOT 305) If expression (zxxu:with ID) 301 matches at offset 10, the logical value of 301 is true while the other patterns’ values are false. Hence, the whole combination’s value is false. Then expression 303 matches at offset 20. Now the values of 301 and 303 are true while the other patterns’ values are still false. In this case, the combination’s value is true, so the combination expression raises a match at offset 20. Finally, expression 305 has matches at offset 30. Now the values of 301, 303 and 305 are true while the other patterns’ values are still false. In this case, the combination’s value is false and no match is raised.

• HS_TUNE_FAMILY_GENERIC:This indicates that the compiled database should not be tuned for any particular target platform.

---hs_runtime.h---

---limits---

The version of Hyperscan used to produce a compiled pattern database must match the version of Hyperscan used to scan with it.

Using the SOM flag entails a number of trade-offs and limitations:

• Reduced pattern support
• Increased stream state: more memory required
• Performance overhead
• Incompatible features: Some other Hyperscan pattern flags can NOT be used in combination with SOM.
• the start offset returned for a match may refer to a point in the stream before the current block being scanned. Hyperscan provides no facility for accessing earlier blocks; if the calling application needs to inspect historical data, then it must store it itself.