Get total key
> dbSize
(integer) 73857708
Get random key
> randomkey
"job_12_02d5b44de7a4af119504abbed239b6f4"
Check the type of the key
> type job_budget_4313410226
hash
Check the expiration time
> ttl job_12_02d5b44de7a4af119504abbed239b6f4
(integer) -1
dump command can dump the value of that key
> dump job_12_02d5b44de7a4af119504abbed239b6f4
"\x04;\x06paused\xc0\x00\x05title\x1cLease Purchase Truck Drivers\x0blast_update\xc2\xc9\xce\xa2X\x05phone\x00\x0cdaily_capped\xc0\x00\teducation\xc0\x02\tsrcseg_id\xc0\x0c\x10excluded_company\xc0\x01\x0eminor_category\xc2\xfa\x9c\t\x00\rquality_score\xc0\x1e\x13top_spot_industries\x00\x06budget\x00\x0ejob_id_mapping\n4310204121\x04tags\x00\bindustry\xc0\x14\x14major_category_score\xc0\x01\x0bis_jobboard\xc0\x01\tsource_id\xc0\x0c\x06is_add\xc0\x00\nsearchtags\x00\x06posted\xc2\xc9\xce\xa2X\x03url\xc3@\x80@\x88\x1fhttp://www.linkup.com/joblisting\x02s/? \x0c\x1f=02d5b44de7a4af119504abbed239b6f\x0c4&utm_source= /\bs2careers`\x17\tmedium=cpc`\x0e\acampaign\xe0\x02(\x01rs\x0bcustomer_id\xc0\x03\x06cities\nDecatur,AL\x0bcampaign_id\xc0\x0c\x05email\x00\areg_cpc\xc0\x00\x06action\x04save\x0faffiliate_group\xc0\x00\acreated\xc2\xc9\xce\xa2X\x06digest 7aefe0914bb80f90e65d4a3f7710ff70\nsegment_id\xc1\xfa\x01\rqq_multiplier\xc0\x00\tj2c_apply\xc0\x00\nbudgetType\bLIFETIME\x06status\aEXPIRED\x0cboost_factor\x0f3.0931627568599\ajobtype\xc0\x01\x03cpc\xc0\x1e\adeduped\x00\x13is_top_spot_preview\xc0\x00\bzipcodes\xc2\x12\x8b\x00\x00\bcpc_high\xc0\x01\nsrccamp_id\xc0\x0c\rjob_reference 02d5b44de7a4af119504abbed239b6f4\acompany\x12Super Service, LLC\x0fmobile_friendly\xc0\x01\x0cdedup_digest 978c2722410a9299ec35229babeeaf37\ncompany_id\xc2R_\x01\x00\bcategory\x00\tstate_ids\xc0\x02\x11quality_sensitive\x00\x14minor_category_score\x030.3\x0cindustry_ids\xc0\x14\x06salary\xc0\x1e\rtop_spot_type\xc0\x00\apool_id\xc0\x0c\x0emajor_category\xc2\xf0\x9c\t\x00\nexperience\xc0\x02\a\x00\xa0\xa0\xbc9\xfd\x84\xa3\""
The tool in here can get the size of one key
https://github.com/sripathikrishnan/redis-rdb-tools
> git clone https://github.com/sripathikrishnan/redis-rdb-tools
> cd redis-rdb-tools/
> sudo python3 setup.py install
> redis-memory-for-key -s test.amazonaws.com -p 6379 job_12_02d5b44de7a4af119504abbed239b6f4
Key job_12_02d5b44de7a4af119504abbed239b6f4
Bytes 5716.0
Type hash
Encoding hashtable
Number of Elements 59
Length of Largest Element 136
Pick up Keys to scan
> scan 77594624 count 10
1) "68157440"
2) 1) "idmapping_4320118968"
2) "job_2465_CB-UnDNml2EZMrepChV7fgIJIrUQWcmv"
3) "job_dedupdigest_f95fa0aea4c4fc1038a191611e69bd7a"
4) "idmapping_4301743608"
5) "job_2465_DkljWeZhGWJ4KpkY023TsSJTq8wRM-xL"
6) "job_budget_4318069707"
7) "job_dedupdigest_d1c8e5ba31218058d0ab96c110f5024f"
"job_budget_4314514161"
9) "job_dedupdigest_4856f59b52ce150c22b92cdd253f3e46"
10) "job_1624_787_416056211"
Scan with Pattern
> scan 0 count 10 match job_*
1) "92274688"
2) 1) "job_budget_4314725990"
2) "job_2465_a1r2Sr6OfBfWgcoEGlDlWumX3Ckx4t3S"
3) "job_4130_2124_951ecb1c50c572cdbcaf0ee261ecd04f_228_i"
4) "job_1627_90433loc459690"
5) "job_1052_8WMXY5c-IUs-T2Y2aPch6xVnhUYPXIEM"
6) "job_budget_4316788589"
7) "job_dedupdigest_6782069fbe5c5e51b742a26b48fe8156"
Finally I write some scripts to measure the memory usage and write a script to scan and clean the Redis.
The major party will be these
if($index % 50 == 0){
$logger->info("processed {$index} / {$count} ...");
}
//fetch key
$key = $redisClient->getRandomKey();
//memory information
$memory = $this->getMemoryUsage($key);
//find the pattern
$pattern = $this->matchPattern($key);
//add the report
if($pattern == '/job_.*/'){
//find job status
$status = $redisClient->getJobStatus($key);
if(empty($status)){
$logger-warning("job {$key} does not have status....");
}
$reports[$pattern . $status]['memory'] = $reports[$pattern . $status]['memory'] + $memory['Bytes'];
$reports[$pattern . $status]['type'] = $memory['Type'];
$reports[$pattern . $status]['count'] = $reports[$pattern . $status]['count'] + 1;
}else{
$reports[$pattern]['memory'] = $reports[$pattern]['memory'] + $memory['Bytes'];
$reports[$pattern]['type'] = $memory['Type'];
$reports[$pattern]['count'] = $reports[$pattern]['count'] + 1;
}
$reports['all']['memory'] = $reports['all']['memory'] + $memory['Bytes'];
$reports['all']['count'] = $reports['all']['count'] + 1;
…snip...
How to fetch memory information
public function getMemoryUsage($key)
{
$redisClient = $this->ioc->getService("redisClient");
exec('redis-memory-for-key -s prod-jobs.pnuura.ng.0001.use1.cache.amazonaws.com -p 6379 ' . $key , $outputs);
$result = array();
foreach ($outputs as $item){
$values = preg_split('/\s+/', $item);
$result[$values[0]] = $values[1];
}
return $result;
}
I scan the keys and clean the related pattern data
public function scanAndCleanExpireJobs($test = true){
$logger = $this->ioc->getService("logger");
$count = 0;
$pattern = 'job_*';
foreach (new Iterator\Keyspace($this->client, $pattern) as $key) {
//$logger->info("key is " . $key);
if(preg_match("/job_dedupdigest_.*/", $key) || preg_match("/job_budget_.*/", $key)){
//ingore
continue;
}
//find job status
$status = $this->client->hget($key, 'status');
if("EXPIRED" == $status){
if($test){
$logger->info("key is " . $key . " status " . $status);
}else{
//delete the expire job
$logger->info("deleting the expire job " . $key . " counting " . $count);
$this->client->del($key);
$count++;
}
}
}
}
Reference:
http://stackoverflow.com/questions/7638542/redis-show-database-size-size-for-keys
https://github.com/snmaynard/redis-audit/blob/master/redis-audit.rb
http://ylw6006.blog.51cto.com/470441/1281548
http://blog.csdn.net/jiangguilong2000/article/details/9313603
http://www.10tiao.com/html/256/201609/2247483748/1.html
https://github.com/antirez/redis-sampler
https://github.com/sripathikrishnan/redis-rdb-tools