Ghidra is a software reverse engineering (SRE) framework
<?xml version="1.0" encoding="utf-8"?>
<book>
<info>
<title>Function ID</title>
</info>
<chapter id="FunctionID">
<title>Function ID</title>
<mediaobject>
<imageobject>
<imagedata condition="noscaling" fileref="images/FIDmatch.png" width="100%" contentwidth="723px" contentdepth="267px" align="center"/>
<imagedata condition="withscaling" fileref="images/FIDmatch.png" width="100%" contentwidth="6in" contentdepth="2.21577in" align="center"/>
</imageobject>
</mediaobject>
<section id="overview">
<title>Overview</title>
<para>
Function ID is an analyzer that performs function identification analysis on a program. For each
function, the analyzer computes a hash of its body and uses this as a key to look up matching
functions in an indexed database of known software. The recovered function name,
and other meta-data about the <emphasis>library</emphasis> it is contained in, is applied to the program.
</para>
<para>
Function ID is suitable for identifying statically linked libraries or other software
where the compiled form of the functions does not change. Because of the hashing strategy,
functions remain identifiable even if the library is relocated during linking.
Larger changes to the compilation process of the library however
will likely prevent successful searches.
Function ID databases are necessarily targeted to a specific processor and optionally to specific
compilers or source languages.
</para>
<para>
Function ID generally runs as part of <emphasis role="bold">Auto Analysis</emphasis> but can
also be initiated at any time
by the user from the <emphasis role="bold">One Shot</emphasis> menu. Ghidra also comes
with a <xref linkend="FunctionIDPlugin"/>, which provides more
control over which databases to apply, and allows users to create and populate their
own databases.
</para>
<para>
By default, Ghidra ships with databases that search for statically linked libraries
from Microsoft Visual Studio for the x86 processor. These have been broken apart into
separate Function ID databases, based on 32-bit or 64-code and the version of Visual Studio.
Within each database, there are a two library variants -- one for debug versions and one for production.
</para>
<sect2 id="hashing">
<title>Hashing</title>
<para>
Function ID works by calculating a cumulative hash over all the machine <emphasis>instructions</emphasis>
that make up the body of a function. For each function, two different 64-bit hashes are computed: a
<emphasis role="bold">full hash</emphasis> and a <emphasis role="bold">specific hash</emphasis>.
Both schemes hash the individual instructions of the function body in address order, but they
differ in the amount of information they include from each instruction.
<informalexample>
<variablelist>
<?dbfo list-presentation="blocks"?>
<varlistentry>
<term><emphasis role="bold">full hash</emphasis></term>
<listitem>
<para>
This hash includes the mnemonic and some of the addressing mode information
from an instruction. Specific register operands are also included as
part of the hash, but the specific value of constant operands are not.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">specific hash</emphasis></term>
<listitem>
<para>
This hash includes everything used in the full hash but may also include
the specific values of any constant operands. A heuristic is employed that
attempts to determine if the constant is not part of an address, in which
case the value is accumulated into the hash.
</para>
</listitem>
</varlistentry>
</variablelist>
</informalexample>
</para>
<para>
Both hashes are used to identify matches in a database. The <emphasis role="bold">full hash</emphasis>
is robust against changes due to linking; the <emphasis role="bold">specific hash</emphasis>
helps distinguish between closely related variants of a function.
</para>
</sect2>
<sect2 id="parentchild">
<title>Parents and Children</title>
<para>
When Function ID examines a function, its parent and child functions are also considered
as a way of disambiguating multiple matches. For example, suppose two functions have identical
sequences of instructions, except they each call to a different subfunction. In this situation,
the full hashes of the functions will be identical, but the system will try to match the hash of one of
the two subfunctions, allowing it to distinguish between the two.
</para>
</sect2>
<sect2 id="libraries">
<title>Libraries</title>
<para>
Within a Function ID database, functions are grouped into <emphasis>libraries</emphasis>,
which are intended to be recognizable named software components
that get linked into larger programs. A <emphasis role="bold">Library</emphasis> has the following meta-data.
<informalexample>
<variablelist>
<?dbfo list-presentation="blocks"?>
<varlistentry>
<term><emphasis role="bold">Name</emphasis></term>
<listitem>
<para>
This is a general descriptive name for the library.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Version</emphasis></term>
<listitem>
<para>
If there is a formal version number for the library, this
field will hold this value as a string.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Variant</emphasis></term>
<listitem>
<para>
Version information that cannot be encoded in the formal
<emphasis>Version</emphasis> field
can be encoded in this field. This is used typically for
<emphasis>compiler settings</emphasis>, which affect Function ID
hashes but don't make sense in a version string describing the
source code.
</para>
</listitem>
</varlistentry>
</variablelist>
</informalexample>
</para>
<para>
Generally, the analyzer is able to report all three fields describing the
library for any function match it finds. In the case where a
database contains multiple versions of the same library, it's common for
a function to match into two or more libraries that differ in their
<emphasis>Version</emphasis> or <emphasis>Variant</emphasis> field. In this case,
the analyzer will still report a single match but will leave off the fields it
couldn't distinguish.
</para>
</sect2>
<sect2 id="singlematches">
<title>Single Matches</title>
<para>
A <emphasis role="bold">Single Match</emphasis> for a function occurs under the following conditions:
<informalexample>
<itemizedlist>
<listitem>The analyzer can narrow down potential matches to a single function name.</listitem>
<listitem>The function does not already have an imported or user-defined name.</listitem>
<listitem>The number of instructions in the function exceeds the <emphasis>instruction count threshold.</emphasis></listitem>
</itemizedlist>
</informalexample>
Even if there are multiple potential matches in the database, the first condition may still hold
because they all share the same base function name.
The second condition does not need to apply if the "Always apply FID labels" option is
toggled on (See <xref linkend="analysisoptions"/>). The number of instructions is computed as
the <emphasis>match score</emphasis> and can include counts of instructions in parent or child functions.
For details about the match score and thresholds, see <xref linkend="scoring"/>.
</para>
<para>
If there is a Single Match, the analyzer will:
<informalexample>
<orderedlist>
<listitem>Apply the function name as a symbol.</listitem>
<listitem>Insert a comment describing the matching library.</listitem>
<listitem>Add a <emphasis>Function ID Analyzer</emphasis> bookmark.</listitem>
</orderedlist>
</informalexample>
Both the inserted comment and bookmark will include the phrase "Single Match".
</para>
</sect2>
<sect2 id="multiplematches">
<title>Multiple Matches</title>
<para>
If the analyzer is not able to narrow down to a single function name,
even after applying all of its disambiguation logic, then the reporting
behavior depends on the remaining match scores. If they are too small
the matches are deemed to be random, and nothing is reported at all.
Otherwise, a <emphasis role="bold">Multiple Match</emphasis> is reported.
In this case, multiple symbols and comments will be inserted, one for
each remaining match, up to an arbitrary limit. All the
comments will contain the phrase "Multiple Matches".
</para>
</sect2>
</section>
<section id="analysisoptions">
<title>Analysis Options</title>
<para>
This analyzer appears under the heading <emphasis role="bold">Function ID</emphasis>
in the dialog listing the standard analyzers whenever the user elects to auto-analyze
a new program upon import, or by selecting <emphasis role="bold">Auto Analyze</emphasis>
under the Code Browser's <emphasis role="bold">Analysis</emphasis> menu. From this dialog, users can
toggle whether the analyzer is active or not, and if
<emphasis role="bold">Function ID</emphasis> is selected and toggled on, the dialog
presents some configurable options for the analyzer.
<informalexample>
<variablelist>
<?dbfo list-presentation="blocks"?>
<varlistentry>
<term><emphasis role="bold">Instruction Count Threshold</emphasis></term>
<listitem>
<para>
This is the <emphasis role="bold">primary threshold</emphasis> a potential
match must exceed in order to be reported by
the analyzer. This defends against
<emphasis>false positives</emphasis> caused by randomly matching
small functions. Roughly,
the score counts the number of instructions in the
function plus instructions in any matching parent or child.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Multiple Match Threshold</emphasis></term>
<listitem>
<para>
In general for a single function, if there are multiple potential matches
all with different names, this is a good sign that the matches are random,
and the analyzer will not report a match. But if the match score exceeds
this threshold, the analyzer will report a <emphasis>Multiple Match</emphasis>.
(See <xref linkend="multiplematches"/>)
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Always Apply FID Labels</emphasis></term>
<listitem>
<para>
If this toggle is on, the analyzer will report matches even if there is
already an imported or user defined name for the function.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Create Analysis Bookmarks</emphasis></term>
<listitem>
<para>
The analyzer will only create bookmarks for matches if this toggle is on.
This does not affect insertion of comments and symbols.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Ignore Database Filters</emphasis></term>
<listitem>
<para>
If this toggle is on, the analyzer will ignore any compiler or
source language filters built into a FID database. Matching will be
attempted when a database targets the same processor as the program,
even if it targets different compilers or source languages.
</para>
</listitem>
</varlistentry>
</variablelist>
</informalexample>
</para>
</section>
<section id="scoring">
<title>Scoring and Disambiguation</title>
<para>
The Function ID analyzer assigns a <emphasis role="bold">match score</emphasis> to
each of the potential matches it discovers in its database. The score is used
both to filter matches which are too small to be significant and to disambiguate
between potential matches.
</para>
<para>
The basic unit of the score is a single matching instruction with no constant
operands, which receives a score of 1.0. Certain instructions, such as calls and
<emphasis>no operation</emphasis> instructions are assigned a score of zero.
Constant operands, in the rare case that they match via the <emphasis>specific hash</emphasis>,
contribute an additional 0.67 units per operand.
</para>
<para>
Once a potential match is discovered, it is assigned a score based on:
<informalexample>
<itemizedlist>
<listitem>Instructions in the body of the function.</listitem>
<listitem>Constant operands that match in the body of the function.</listitem>
<listitem>Instructions in the body of any child function that also has a match.</listitem>
<listitem>Instructions in the body of any parent function that also has a match.</listitem>
</itemizedlist>
</informalexample>
Once scores are assigned, potential matches are filtered based on the
<emphasis>instruction count threshold</emphasis> (See <xref linkend="analysisoptions"/>).
This helps prevent small functions that randomly match database entries
from being reported. Note however that a small
function can still be correctly reported if its parent or child functions also have matches,
increasing its overall score.
If there are still more than one potential match, the highest assigned score is
used to filter out matches with lower scores.
</para>
<sect2 id="matchingfunction">
<title>Matching Function Names</title>
<para>
If there are still multiple potential matches once thresholds have been applied to the
match scores, the remaining matches will be grouped based on function names. If
two potential matches share the same function name, they are grouped together.
If the remaining matches can all be placed into a group sharing a single name, the result will
still be reported as a "Single Match".
</para>
<para>
Function names are considered to match if their <emphasis>base</emphasis> names match.
The base name is obtained by stripping off any namespace from the symbol plus any initial underscores.
If the name is mangled, an attempt is made to demangle it first, then namespace and
parameter information is stripped.
</para>
</sect2>
</section>
</chapter>
<chapter id="FunctionIDPlugin">
<title>Function ID Plug-in</title>
<para>
The Function ID Plug-in allows users to create new (.fidb) databases.
A Function ID database is a collection of meta-data describing software libraries
and the functions they contain. It is searchable via a hash computed over the
body of unknown functions (See <xref linkend="FunctionID"/>). The databases are self-contained
and can be shared among different users.
Using this plug-in, databases can be created, attached, and detached
from an active Code Browser, and a database
can be populated with function hashes from programs in the current Ghidra project.
</para>
<para>
A Function ID database may hold many distinct libraries. Within a single library, all
functions must come from the same <emphasis>processor model</emphasis> as determined
by Ghidra's Language ID. Ideally, functions should come from a single software component,
all compiled using the same compiler and settings.
</para>
<para>
Functions for a single library can be ingested from multiple files, usually from a series
of <emphasis>object</emphasis> files analyzed within a Ghidra repository. If the library
is spread across multiple files, accurate symbol information is necessary
to properly compute parent/child relationships.
</para>
<section id="enableplugin">
<title>Enabling the Plug-in</title>
<para>
All plug-in functionality is accessible from the <emphasis role="bold">Function ID</emphasis>
menu, under the main <emphasis role="bold">Tools</emphasis> menu.
In order to access this menu, the tool must be configured to include the
Function ID plugin. To do this, from the Code Browser select
<informalexample>
<emphasis role="bold">File -> Configure</emphasis>
</informalexample>
Then click on the <emphasis>Configure</emphasis> link under the
<emphasis role="bold">Ghidra Core</emphasis> section and check the box
next to "FidPlugin".
</para>
</section>
<section id="pluginfunctionality">
<title>Plug-in Functionality</title>
<para>
The Function ID plug-in provides the following actions under <emphasis role="bold">Tools -> Function ID</emphasis>.
</para>
<sect2 id="chooseactivemenu">
<title>Choose active FidDbs...</title>
<para>
This brings up a dialog that allows the user to select which Function ID databases
are active. By default, Ghidra ships with a set of databases and they are all initially
active.
</para>
<mediaobject>
<imageobject>
<imagedata condition="noscaling" fileref="images/ChooseActiveFidDbs.png" width="100%" contentwidth="400px" contentdepth="400px" align="center"/>
<imagedata condition="withscaling" fileref="images/ChooseActiveFidDbs.png" width="100%" contentwidth="4in" contentdepth="4in" align="center"/>
</imageobject>
</mediaobject>
<para>
Once a database has been deactivated, it will no longer be used for matches in
subsequent analysis. The selections for which databases are active are saved
as a preference on a per-user basis.
</para>
</sect2>
<sect2 id="createemptyfid">
<title>Create new empty FidDb...</title>
<para>
This brings up a file chooser dialog that allows the user to create a new
Function ID database. This cannot be located under the Ghidra install directory
root, because Ghidra considers files under the root read-only. We recommend
ending this database with the extension .fidb for consistency, although not
strictly necessary. Newly created databases are attached
(which means "known" for the purposes of tracking) and initially active,
even though they contain no match entries.
</para>
</sect2>
<sect2 id="attachfid">
<title>Attach existing FidDb...</title>
<para>
This brings up a file chooser dialog that allows the user to attach an
existing Function ID database. This cannot be located under the Ghidra
install directory root, because Ghidra considers files under the root
read-only. Attached databases are saved in the user preference system,
and retain their active status across sessions of Ghidra.
</para>
</sect2>
<sect2 id="detachfid">
<title>Detach attached FidDb...</title>
<para>
This brings up a dialog that allows the user to detach an already-attached
Function ID database. None of the databases delivered with Ghidra can be detached;
they can only be deactivated (see <xref linkend="chooseactivemenu"/>). Detaching a
database removes it from use in searching, and also causes the user preference
system to forget about the existence of this database.
</para>
<mediaobject>
<imageobject>
<imagedata condition="noscaling" fileref="images/DetachAttachedFidDb.png" width="100%" contentwidth="409px" contentdepth="131px" align="center"/>
<imagedata condition="withscaling" fileref="images/DetachAttachedFidDb.png" width="100%" contentwidth="4in" contentdepth="1.2812in" align="center"/>
</imageobject>
</mediaobject>
</sect2>
<sect2 id="populatedialog">
<title>Populate FidDb from programs...</title>
<para>
Once a database has been created (or attached), the user may populate it
with hash values from a set of programs in the current project. Choosing
this option brings up a dialog where users enter the information needed
to populate the database.
</para>
<mediaobject>
<imageobject>
<imagedata condition="noscaling" fileref="images/PopulateFidDbFromPrograms1.png" width="100%" contentwidth="422px" contentdepth="320px" align="center"/>
<imagedata condition="withscaling" fileref="images/PopulateFidDbFromPrograms1.png" width="100%" contentwidth="4in" contentdepth="3.033175in" align="center"/>
</imageobject>
</mediaobject>
<sect3 id="dialogfields">
<title>Dialog Fields</title>
<para>
<informalexample>
<variablelist>
<?dbfo list-presentation="blocks"?>
<varlistentry>
<term><emphasis role="bold">Fid Database</emphasis></term>
<listitem>
<para>
Pick the database to populate. Users must select from attached databases that are writable.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Library Family Name</emphasis></term>
<listitem>
<para>
The name of the library being ingested. This is the identifier that will be inserted
as part of the comment when a match is found for functions in this library.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Library Version</emphasis></term>
<listitem>
<para>
The formal version string for the library. This is frequently the
<emphasis><major>.<minor>.<patch></emphasis>
syntax but can be anything provided to distinguish between different versions of the same library.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Library Variant</emphasis></term>
<listitem>
<para>
Any other string for distinguishing libraries that doesn't fit in the formal
<emphasis>Version</emphasis> string. This frequently holds compiler settings.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Base Library</emphasis></term>
<listitem>
<para>
This is an optional setting for cross-linking with an existing library already
ingested in the database. In the event that the user wants to incorporate parent/child
relationships into this library from another related library, they can set this option
to point at the other library. The library must already be ingested (from a previous use
of this command) into the same database. The Function ID ingest process will match parents
and children across the libraries via function symbols.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Root Folder</emphasis></term>
<listitem>
<para>
This specifies the set of programs from which the new library
will be populated. The user can select any subfolder in the current Ghidra project. The
ingest process will recursively search through all programs beneath this folder and,
for each program, will collect any functions it contains.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Language</emphasis></term>
<listitem>
<para>
This is the required 4-field Ghidra Language ID (i.e. <emphasis>x86:LE:64:default</emphasis>)
specifying exactly what processor the new library will contain. While scanning during ingest,
any program that does not match this program model will be automatically skipped.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Compiler Specs</emphasis></term>
<listitem>
<para>
This is an optional comma separated list of compiler names. If at least one name is listed,
the new library will apply only to programs with a matching compiler specification.
Programs without a matching compiler specification will be filtered out during the
ingest process and, by default, cannot query against the library.
Each name listed must match a compiler specification associated with the selected Language ID.
If no names are listed, any compiler specification is considered matching for the
new library, and no programs will be filtered.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Source Languages</emphasis></term>
<listitem>
<para>
This is an optional comma separated list of names that are compared against a
program's "Source Languages" property. If at least one name is listed,
the new library will apply only to programs with a matching source language.
Programs that do not match will be filtered out during the
ingest process and, by default, cannot query against the library.
If no names are listed, any source language is considered matching for the
new library, and no programs will be filtered.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Common Symbol File</emphasis></term>
<listitem>
<para>
This is an optional parameter that provides a list of common function symbols to
the ingest process. The parameter, if present, is a path to a text file that contains
the list of function symbols, one per line. The ingest process excludes these functions
from consideration as disambiguating child functions. (See <xref linkend="falsepositives"/>)
</para>
</listitem>
</varlistentry>
</variablelist>
</informalexample>
</para>
</sect3>
<para>
Once all fields have been filled, clicking <emphasis role="bold">OK</emphasis> causes all selected
functions to be ingested for the new library. Depending on the number of programs and
functions selected, this process may take some time. Upon completion, the process will present
a summary window, containing ingest statistics and an ordered list of functions that were
most commonly called within the library. This list can be used to create a
<emphasis>Common Symbol File</emphasis> tailored for the library.
</para>
</sect2>
</section>
<section id="preparelibraries">
<title>Preparing Libraries for a Function ID Database</title>
<sect2 id="programlocation">
<title>Location of Programs</title>
<para>
All functions going into a single Function ID <emphasis>Library</emphasis> must already be imported and analyzed
somewhere within a single Ghidra repository (shared or non-shared). Multiple libraries contained within
the same database can be ingested in different phases, but a single library must be written to the database
in a single pass. The ingest dialog (see <xref linkend="populatedialog"/>) specifies a single subfolder as
the root for the library. The process acts recursively, so there can be additional directory hierarchy under the root,
but all programs to be included in the library must be under the one root.
</para>
</sect2>
<sect2 id="analysis">
<title>Analysis</title>
<para>
All programs must be analyzed enough to have recovered the bodies of all the functions that are to be included
in the library. Generally, the easiest way to accomplish this is to run Ghidra's default auto-analysis.
If functions are spread across multiple programs, as is typically the case, users can run the
Ghidra's <code>analyzeHeadless</code> command to analyze across the whole set. However, take note below of some
of the modifications to the default analysis that may be necessary to get good Function ID results.
</para>
<para>
Every function to be included must have a <emphasis>non-default</emphasis> function name assigned.
Function ID uses a function's <emphasis>primary symbol</emphasis> for the name.
Symbols are typically imported from debug information, but any method for assigning
names, script based or manual, will work. Any function that still has its default name,
<code>FUN_00....</code>, currently will not be ingested.
</para>
<para>
When performing auto-analysis in preparation for ingest, its best to disable the
<emphasis role="bold">Function ID</emphasis> analyzer (and the
<emphasis role="bold">Library Identification</emphasis> analyzer as well)
in order to avoid cross contamination from different databases. If the function symbols
are <emphasis>mangled</emphasis>, be sure to turn off the <emphasis role="bold">Demangler</emphasis>
analyzer. This lets the future database apply the raw mangled symbol to new programs during analysis, which
lets their <emphasis role="bold">Demangler</emphasis> analyzer pass run with complete information.
</para>
<para>
For an example of these sorts of modifications to the analysis process, see the file:
<informalexample>
<filename>Features/FunctionID/ghidra_scripts/FunctionIDHeadlessPrescript.java</filename>
</informalexample>
This is designed to be passed to the <code>analyzeHeadless</code> command as a pre-script option.
</para>
</sect2>
</section>
<section id="falsepositives">
<title>False Positives</title>
<para>
A <emphasis role="bold">false positive</emphasis> in the context of Function ID is a
function that is declared as a match by the analyzer but has the incorrect symbol applied.
As with any classification algorithm, it is generally not possible to eliminate this kind
of error completely, but with Function ID there are some mitigation strategies.
</para>
<sect2 id="causes">
<title>Causes</title>
<para>
False positives for the most part only happen with small functions.
There are two related causes with Function ID:
<informalexample>
<variablelist>
<?dbfo list-presentation="blocks"?>
<varlistentry>
<term><emphasis role="bold">Tiny Functions</emphasis></term>
<listitem>
<para>
If a function consists of only a few instructions, it can be
matched <emphasis>randomly</emphasis> if there are enough entries
in the database. The fewer operations a function performs, the
more likely an unrelated function is to do those exact same things.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Code Idioms</emphasis></term>
<listitem>
<para>
A set of functions that are effectively
identical can have different names and be used in unrelated contexts.
Functions such as <emphasis>destructors</emphasis> are typical: one might
check that a particular structure field is non-zero, and then pass that
field to <code>free</code>. Another destructor may
perform the exact same sequence, but was designed for a completely unrelated structure.
</para>
</listitem>
</varlistentry>
</variablelist>
</informalexample>
</para>
<para>
In either case, Function ID can apply a symbol that is misleading for the analyst.
</para>
</sect2>
<sect2 id="mitigation">
<title>Mitigation via Threshold</title>
<para>
All mitigation strategies, to some extent, trade-off false positives for
<emphasis role="bold">false negatives</emphasis>, which are functions that should have
been reported by the Function ID analyzer, but aren't (because of some threshold or strategy).
</para>
<para>
Most false positives by far are due to tiny functions. Function ID minimizes these via the
<emphasis>instruction count threshold</emphasis>. Potentially matching functions with
too few instructions that don't exceed this threshold will simply not be reported by the analyzer.
</para>
<para>
For users experiencing too many false positives, the instruction count threshold
is the easiest thing to adjust. It is fully controllable
by the user as an Analysis option (See <xref linkend="analysisoptions"/>), and increasing
it will directly reduce the false positive rate, at the expense of missing some <emphasis>true</emphasis>
matches whose scores now fall below the threshold.
</para>
</sect2>
<sect2 id="specialmitigate">
<title>Specialized Mitigation</title>
<para>
The default instruction count threshold is a good starting point for any new database,
generally striking a reasonable balance limiting false positives without
eliminating too many true matches. But even for an optimal threshold,
there may be a small handful of functions in the new database (usually <emphasis>Code Idioms</emphasis>)
that exceed the threshold and repeatedly cause the wrong label to be placed. Instead of increasing
the threshold to filter out <emphasis>all</emphasis> functions with these higher scores, it is
possible to turn on one of several mitigation strategies that target the offending database entries
directly. These strategies include:
<informalexample>
<variablelist>
<?dbfo list-presentation="blocks"?>
<varlistentry>
<term><emphasis role="bold">Force Specific</emphasis></term>
<listitem>
<para>
If this is set on an entry, the specific hash must match before the
system will consider the entry as a potential match. This is useful when
a code idiom contains a known constant that the full hash would usually miss.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Force Relation</emphasis></term>
<listitem>
<para>
This is probably the most useful specialized strategy. It forces
at least one parent or child match to be present before the system considers the base
function as a potential match. So even if an idiom is big, this forces a search
for an additional confirmation.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Auto Fail</emphasis></term>
<listitem>
<para>
This is a strategy of last resort. If an obnoxious code idiom can't be
eliminated any other way, this forces the particular database entry to
never be considered as a match.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Auto Pass</emphasis></term>
<listitem>
<para>
This strategy is different from the others, in that it applies to function
entries whose scores are slightly too <emphasis>low</emphasis>.
If a low scoring function has an instruction sequence that is deemed unique enough,
this strategy causes any potential match to automatically pass the threshold.
This provides an alternative to lowering the instruction count threshold to include
a particular function.
</para>
</listitem>
</varlistentry>
</variablelist>
</informalexample>
</para>
<para>
These strategies can all be toggled for <emphasis>individual function records</emphasis> in the database.
To do this manually from the Code Browser, the user needs to search for the specific records they want to
change using the <emphasis>Debug Search Window</emphasis> and then make changes from its
<emphasis>Result Window</emphasis>. For details see <xref linkend="debugsearch"/>.
</para>
<para>
Strategies can also be toggled by running a Ghidra script.
Within a script the basic instruction sequence looks like:
<informalexample>
<programlisting>
FidFileManager fidFileManager = FidFileManager.getInstance();
List<FidFile> allKnownFidFiles = fidFileManager.getFidFiles();
// Choose a modifiable database from the list
...
// Open a specific database
FidDB modifiableFidDB = fidFile.getFidDB(true);
// Toggle strategies based on the full hash of the function(s)
modifiableFidDB.setAutoFailByFullHash(0x84d01243dfb8b9cbL, true);
modifiableFidDB.setForceRelationByFullHash(0x4e0920960b48ae7eL, true);
modifiableFidDB.setForceSpecificByFullHash(0x5ef2f47ee7151243L, true);
modifiableFidDB.setAutoPassByFullHash(0x96a4a6fd5694523bL, true);
...
// Save and close the database
modifiableFidDB.saveDatabase("comment", monitor);
modifiableFidDB.close();
</programlisting>
</informalexample>
</para>
<para>
FunctionID hashes for specific functions can be obtained with the <code>FIDHashCurrentFunction</code>
script.
</para>
</sect2>
</section>
</chapter>
<chapter id="FunctionIDDebug">
<title>Function ID Debug Plug-in</title>
<para>
The Function ID Debug Plug-in allows users to inspect the individual records in a Function ID database.
This functionality is generally only useful to users building their own databases. Users can look up
individual records based on name or hash, but currently the
plug-in is not capable of modifying records. The only exceptions are the <xref linkend="specialmitigate"/>
strategies, which can be modified using the <xref linkend="debugsearch"/>.
</para>
<section id="enabledebug">
<title>Enabling the Plug-in</title>
<para>
The Debug Plug-in adds options to the <emphasis role="bold">Function ID</emphasis> menu,
under the Code Browser's main <emphasis role="bold">Tools</emphasis> menu. These options are in
addition to those introduced by the <xref linkend="FunctionIDPlugin"/>, which uses the same menu.
In order to access the Debug options, the plug-in must be enabled. To do this, from the Code Browser select
<informalexample>
<emphasis role="bold">File -> Configure</emphasis>
</informalexample>
Then click on <emphasis>Configure</emphasis> link under the <emphasis role="bold">Developer</emphasis>
section and check the box next to "FidDebugPlugin".
</para>
</section>
<section id="debugfunctionality">
<title>Plug-in Functionality</title>
<para>
The Function ID Debug Plug-in introduces the following actions to the
<emphasis role="bold">Tools -> Function ID</emphasis> menu.
</para>
<sect2 id="readonly">
<title>Create Read-only Database</title>
<para>
Users can convert the read/write (.fidb) database into the a read-only (.fidbf) form. This is
the more efficient final form used directly by the Function ID analyzer. The .fidbf form is
uncompressed on disk and the analyzer can use it directly, where the .fidb form must be converted
before use.
</para>
</sect2>
<sect2 id="debugsearch">
<title>Debug Search Window</title>
<para>
This action brings up a Search dialog for the currently active Function ID databases. The text entry
fields correspond to the individual fields in a <emphasis>function record</emphasis> that can
be searched. If the variant <emphasis role="bold">Debug Search Window (Current Function)</emphasis>
is invoked, the same dialog is brought up, but the <emphasis>Full Hash</emphasis> and the
<emphasis>Specific Hash</emphasis> fields are pre-populated with hashes corresponding to the
function at the current address.
</para>
<para>
A search is initiated for a specific search field by entering a value and then hitting
the <command>RETURN</command> key, with the cursor and focus still in the desired field.
</para>
<mediaobject>
<imageobject>
<imagedata condition="noscaling" fileref="images/FIDSearch.png" width="100%" contentwidth="315px" contentdepth="263px" align="center"/>
<imagedata condition="withscaling" fileref="images/FIDSearch.png" width="100%" contentwidth="4in" contentdepth="3.3397in" align="center"/>
</imageobject>
</mediaobject>
<sect3 id="searchfields">
<title>Search Fields</title>
<para>
<informalexample>
<variablelist>
<?dbfo list-presentation="blocks"?>
<varlistentry>
<term><emphasis role="bold">Function ID</emphasis></term>
<listitem>
<para>
This is the internal row ID for function records in the database.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Name</emphasis></term>
<listitem>
<para>
This searches through function names. Searches here will match any record
whose name contains the search string.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Domain Path</emphasis></term>
<listitem>
<para>
This searches through domain paths. A <emphasis role="bold">domain path</emphasis>
is the file path, relative to the project root, of a program containing the
function described by a particular record. Searches here will match any
record whose domain path contains the search string.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">FH</emphasis></term>
<listitem>
<para>
This searches for records matching the <emphasis>full hash</emphasis>. The
text field expects a 64-bit value. The hash can be entered as a hexadecimal string
by prepending with "0x".
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">XH</emphasis></term>
<listitem>
<para>
This searches for records matching the <emphasis>specific hash</emphasis>. The
text field expects a 64-bit value. The hash can be entered as a hexadecimal string
by prepending with "0x".
</para>
</listitem>
</varlistentry>
</variablelist>
</informalexample>
</para>
</sect3>
<sect3 id="resultwindow">
<title>Result Window</title>
<para>
Invoking a search will bring up the <emphasis>Result Window</emphasis>, presenting a row for
each matching function record. Columns list properties of the function and correspond
to the search fields described above. In addition to these, each record/row lists a few
other columns:
<informalexample>
<variablelist>
<?dbfo list-presentation="blocks"?>
<varlistentry>
<term><emphasis role="bold">Library</emphasis></term>
<listitem>
<para>
This is the library containing the function
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Code Unit Size</emphasis></term>
<listitem>
<para>
This is the number of (scoring) instructions in the function's body.
See <xref linkend="scoring"/>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Spec. + Size</emphasis></term>
<listitem>
<para>
This is the number of distinct constant operands fed into
the specific hash.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Warn</emphasis></term>
<listitem>
<para>
This lists any special properties that have been toggled for the particular record. The
column is presented as a string of single character codes, corresponding to each possible property.
Properties include the mitigation strategies described in <xref linkend="specialmitigate"/>.
<itemizedlist>
<listitem>F - Auto Pass: The record automatically fails.</listitem>
<listitem>P - Auto Fail: The record automatically passes.</listitem>
<listitem>S - Force Specific: Specific hash must match for record to be considered.</listitem>
<listitem>R - Force Relation: A parent or child must match for record to be considered.</listitem>
<listitem>U - The function body was unterminated (analysis error).</listitem>
</itemizedlist>
</para>
</listitem>
</varlistentry>
</variablelist>
</informalexample>
</para>
<sect4 id="editmenu">
<title>Edit Menu</title>
<para>
The <emphasis>Result Window</emphasis> supports a small number of actions under the
<emphasis role="bold">Edit</emphasis> menu that allow the user to change the
mitigation strategies for a read/write database. Strategies are described in
<xref linkend="specialmitigate"/>. When a menu action is invoked, all records
displayed in the current Result Window are affected. Currently there is no way
to select a subset of records to effect. The actions all set or clear a specific
strategy.
<informalexample>
<variablelist>
<?dbfo list-presentation="blocks"?>
<varlistentry>
<term><emphasis role="bold">Set/Clear auto-fail</emphasis></term>
<listitem>
<para>
Toggle the <emphasis>Auto Fail</emphasis> strategy.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Set/Clear auto-pass</emphasis></term>
<listitem>
<para>
Toggle the <emphasis>Auto Pass</emphasis> strategy.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Set/Clear force-specific</emphasis></term>
<listitem>
<para>
Toggle the <emphasis>Force Specific</emphasis> strategy.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Set/Clear force-relation</emphasis></term>
<listitem>
<para>
Toggle the <emphasis>Force Relation</emphasis> strategy.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">Save changes</emphasis></term>
<listitem>
<para>
Changes made using the <emphasis role="bold">Edit</emphasis> menu are
not immediately saved back to the underlying database until this
action is invoked.
</para>
</listitem>
</varlistentry>
</variablelist>
</informalexample>
</para>
</sect4>
</sect3>
</sect2>
<sect2 id="tableviewer">
<title>Table Viewer</title>
<para>
This invokes an extremely low-level view into the underlying tables that back a
Function ID database. It can be invoked on any attached database. A window is brought up
that lists individual records in one table. A drop-down menu allows the user to switch
between the different underlying tables. Most of the columns contain row keys and don't
present readable values. The only meaningful table is likely to be the
<emphasis>Libraries Table</emphasis> which will list each library making up the database.
</para>
</sect2>
</section>
</chapter>
</book>