Derived Field Best Practices

Overview

We'll be discussing how to think about Derived Fields and leverage them to their fullest extent. We'll be assuming that you know the basics, so if you haven't read how Derived Fields work   yet, you might want to before reading this.

At a high level, Derived Fields are meant to provide a single value you can rely on to search across all of your scanner targets as opposed to having to have all of your users remember the nuances of each scanner, such as Scanner X places an FQDN whereas scanner Y places as Hostname field. To most users thats just a Name field, so being able to have a derived field like Target Name helps tremendously.

Logic

Let's talk about the logic for Derived Fields first so we can then think of strategies of how to best use them. 

The goal of a derived field ruleset is to ideally always end with a value for the derived field. Each rule is a series of cases, or If / Else statements where the condition is analogous to:

• IF <field value> exists then use as the value of Derived Field.
• ELSE check to see if the next <field value> exists.
• UNTIL at least one value is found or all rules have been processed. 
• Where <field value> is any value that is not "" or null. 

Simple Example

Let's go with an easy situation where you only have one integration, let's say it's Qualys but it could be Tenable or some other integration. 

You ultimately want to be able to search for a host / asset by its name or maybe by its IP address. But depending on the target type (ie. router, workstation, or server) or the type of scan Qualys is using (authenticated vs unauthenticated, or external vs. internal) there may or not be targets with a hostname field value, or an fqdn field value, or an IP Address only. 

Maybe you can figure this nuance out but as you onboard hundreds of users they may not know. What we know for sure is we want to query by a Name or by a Location. 

So we can configure the derived field for Target Name to have a set of rules based on the types of targets and scans I know we use in our network.

1. I know that our users and others internally reference hosts by their FQDN so ideally all Target Names would be an FQDN. 
2. However, I also know that Qualys is configured to scan routers and other networking devices so in those situations I'm ok if the Target Name is the IP address. 
3. I also know that in some cases we may not have an FQDN, so I would prefer to have the value come from something like a hostname or maybe the netbios.
4. Or maybe in my particular organization I trust the CMDB more than I do the scanner and I would rather use the name provided by the CMDB via a tag in NopSec. 

We essentially just ran through the logic for Target Name, now we just need to place that as a rule set.

1. Go to the Integration Settings page and click on the Derived Fields tab
2. Find the Target Name field and open it's collapsible view by clicking on the downward arrow.
3. Add the first rule by looking for the field FQDN
4. Then add another rule looking for Hostname
5. Then add another rule looking for Netbios
6. Then add another rule looking for a Tag
7. Then add the final rule looking for an IP Address

Now for every target that is synced NopSec will run through this ruleset and stop when a value is found. 

You would follow this exercise for all fields that are relavant to you.

• What have we configured the scanner to scan?
• What are the nuances of this scanner? Where does it place the most important value?
• Are there alternatives that would satisfy the need if other values are empty?

Normalization across Scanners

Derived fields are powerful querying fields as you can leverage them to maintain consistency across scanners. By themselves they are not required but they are very helpful. 

If you have scanners scanning the same resources then you can leverage Derived fields to improve your querying by normalizing the value types/formats across scanners. 

For instance, lets say your organization speaks about a host via an FQDN format. You'd like to have all targets have a Target Name with a value formatted as an FQDN. The simplest thing to do is to have a ruleset where the first rule is looking for the existence of an FQDN field. 

Done. 

Harder Example

But let's say you want to have an Organization field associated with all targets for easier reporting, so you want to configure the Target Organization derived field. 

When you look at your scanners you identify that your organization is leveraging Tags to store this information, however, because each scanner has different functionality the way each target is being tagged by those platforms may differ greatly or subtly. 

Let's use the following two targets as an example:

In this example Target A has two Tags and Target B has three. When looking at them they kind of look similar but they're not exactly the same. In Target A the platform returns tags as a Key Value pair where the users in that platform can configure and specify the Key of the tag. While in Target B the scanner is not providing a Key Value pair but instead just a string, in this situation NopSec attempts to normalize towards a Key Value object in the database by creating a Key of Scanner Group.

So if we wanted Target Organization to end up as

• Target Organization = Product

We'd have to create the following derived rules:

1. IF Tag Key is exact Org
   1. Note: when using the Tag rules, the value returned when a match is found is the Tag Value of the tag.
2. ELSE IF Tag Value is exact Product
3. ELSE IF Tag Value contains Prod

With the above rule set, the final value for each Target would be Target Organization = Product.

Why did we add the third rule? Well in case the use of the Product value in Qualys is not consistent but rather the use of Org - Prod is then we can capture that use case. In this situation the value for Target Organization on Target B would be Org - Prod. 

This isn't the best for your end users as they would have to be told to do queries with contains "Prod" vs "Product" to have the best chance of finding the correct targets. 

Today you would have to resolve this issue in your Scanner Tagging practices or manually tag your targets within NopSec using our Tag File Upload process. 

We are considering adding  a way to allow you to map the value of a field to a normalized value such that in this situation you would have the rule as:

• ELSE IF Tag Value contains Prod
  • IF Org - Prod then Product

If this was available the final result of Target Organization then would still be Product. If this would be helpful for you let us know and we can prioritize it.

Thought Process

Hopefully the above examples give you the context to start thinking of how to use Derived Fields to your advantage. We would like to continue expanding the number of derived fields and the logic capabilities to enable the use above but also others like potentially using Regex named groups. 

Remember, the goal is to think about the rest of your users and helping them in their day to day querying. So think:

• How do we as an organization reference this field?
• Where do we place that value in our process? ie. Where's our System of Record for this?
• How do the scanners differ in implementation for this value?
• How can I create rules between both scanners such that the end is more likely to be the same value? 
• Or what process do I need to change to help me reach that goal.