osquery An Operating System Instrumentation Framewor

catalog

1. Getting Started
2. install guide for OS X and Linux
3. Features Overview
4. Logging
5. query example

 

1. Getting Started

osquery is an operating system instrumentation framework for OS X and Linux. The tools make low-level operating system analytics and monitoring both performant and intuitive.
osquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. With osquery, SQL tables represent abstract concepts such as

1. running processes
2. loaded kernel modules
3. open network connections
4. browser plugins
5. hardware events or file hashes 

Relevant Link:

http://osquery.readthedocs.org/en/stable/
http://www.oschina.net/p/osquery
http://www.infoq.com/cn/news/2014/10/osquery-facebook-sql

 

2. install guide for OS X and Linux

0x1: Ubuntu Trusty 14.04 LTS

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C9D8B80B
sudo add-apt-repository "deb [arch=amd64] https://osquery-packages.s3.amazonaws.com/trusty trusty main"
sudo apt-get update
sudo apt-get install osquery

0x2: CentOS/RHEL 6.6

sudo rpm -ivh https://osquery-packages.s3.amazonaws.com/centos6/noarch/osquery-s3-centos6-repo-1-0.0.noarch.rpm
sudo yum install osquery

0x3: Optional: Kernel driver

osquery does not require a kernel driver currently. There are medium priority plans to extend table data collection into the kernel as well as use kernel frameworks to protect the daemon and log data.

Relevant Link:

http://osquery.readthedocs.org/en/stable/installation/install-linux/

 

3. Features Overview

1. osqueryd
The high-performance and low-footprint distributed host monitoring daemon, osqueryd, allows you to schedule queries to be executed across your entire infrastructure. The daemon takes care of aggregating the query results over time and generates logs which indicate state changes in your infrastructure. You can use this to maintain insight into the security, performance, configuration, and state of your entire infrastructure. osqueryd's logging can integrate into your internal log aggregation pipeline, regardless of your technology stack, via a robust plugin architecture.

2. osqueryi
The interactive query console, gives you a SQL interface to try out new queries and explore your operating system. With the power of a complete SQL language and dozens of useful tables built-in, osqueryi is an invaluable tool when performing incident response, diagnosing an systems operations problem, troubleshooting a performance issue, etc.

3. osquery 
osquery is cross platform. Even though osquery takes advantage of very low-level operating system APIs, you can build and use osquery on Mac OS X, Ubuntu, Cent OS and other popular enterprise Linux distributions

4. plugin architecture
To assist with the rollout process, the osquery user guide has detailed documentation on internal deployment. osquery was built so that every environment specific aspect of the toolchain can be hot-swapped at run-time with custom plugins. Use these interfaces to deeply integrate osquery into your infrastructure if one of the several existing plugins don not suit your needs

总体来说，osquery的特点如下

1. 是采用了定时采样的方式收集主机上相关信息，适合在大量的分布式集群上部署agent，在一个中心server上部署analysis进行数据集中分析
2. osquery使用了用户态的系统API获取系统上相关信息，包括
    1) running processes(运行中进程)
    2) loaded kernel modules
    3) open network connections(网络外连)
    4) browser plugins
    5) hardware events or file hashes 
3. 对于入侵检测有帮助的字段有"running processes"、"open network connections"，但是osquery的采样方式觉得它获取的不是实时的准确数据，因为恶意的"running processes"和"open network connections"都有可能是瞬间发生的
4. osquery有一个beta版的内核态采样模块，但是处于unstable状态，工作较不稳定，而且工作方式依然是定时采样，而不是实时的hook

Relevant Link:

https://osquery.readthedocs.org/en/stable/introduction/overview/

 

4. Logging

The osquery daemon uses a default filesystem logging plugin. Like the config, output from the filesystem plugin is written as JSON. Results from the query schedule are written to /var/log/osquery/osqueryd.results.log.
There are two types of logs:

1. Status logs (info, warning, error, and fatal)
2. Query schedule results logs

osquery的技术思路就是在本地进行定时的采样(周期性地调度本地的默认SQL语句)，获取机器信息，将全部事件信息存储在本地，并将log数据进行封装，对外封装了一个SQL Query查询接口，接收来自中心sever的查询请求
osquery的特点就是不需要通过网络从agent收集log，而通过本地存储日志，而中心server下发轻量级的sql获取所需要的信息，从而减小了网络传输的消耗和中心server分析和处理庞大log的消耗

Relevant Link:

https://osquery.readthedocs.org/en/stable/deployment/logging/

 

5. query example

1. List the the users:
SELECT * FROM users;

2. Check the processes that have a deleted executable:
SELECT * FROM processes WHERE on_disk = 0;

3. Get the process name, port, and PID, which are listening on all interfaces:
SELECT DISTINCT process.name, listening.port, process.pid
FROM processes AS process
JOIN listening_ports AS listening ON process.pid = listening.pid
WHERE listening.address = '0.0.0.0';

4. Find every OS X LaunchDaemon that launches an executable and keeps it running:
SELECT name, program || program_arguments AS executable
FROM launchd
WHERE
  (run_at_load = 'true' AND keep_alive = 'true')
AND
  (program != '' OR program_arguments != '');

5. Check for ARP anomalies from the host's perspective:
SELECT address, mac, count(mac) AS mac_count
FROM arp_cache GROUP BY mac
HAVING count(mac) > 1;

6. Alternatively, you could also use a SQL sub-query to accomplish the same result:
SELECT address, mac, mac_count
FROM
  (SELECT address, mac, count(mac) AS mac_count FROM arp_cache GROUP BY mac)
WHERE mac_count > 1;

Relevant Link:

https://github.com/facebook/osquery

 

Copyright (c) 2015 LittleHann All rights reserved