This file documents awk, a program that you can use to selectparticular records in a file and perform operations upon them.
Copyright © 1989, 1991, 1992, 1993, 1996, 1997, 1998, 1999,2000, 2001, 2002, 2003, 2004, 2005, 2007, 2009, 2010, 2011Free Software Foundation, Inc.
This is Edition 4 of GAWK: Effective AWK Programming: A User's Guide for GNU Awk,for the 4.0.0 (or later) version of the GNUimplementation of AWK.
Permission is granted to copy, distribute and/or modify this documentunder the terms of the GNU Free Documentation License, Version 1.3 orany later version published by the Free Software Foundation; with theInvariant Sections being “GNU General Public License”, the Front-Covertexts being (a) (see below), and with the Back-Cover Texts being (b)(see below). A copy of the license is included in the section entitled“GNU Free Documentation License”.
FS
from the Command Linegetline
getline
with No Argumentsgetline
into a Variablegetline
from a Filegetline
into a Variable from a Filegetline
from a Pipegetline
into a Variable from a Pipegetline
from a Coprocessgetline
into a Variable from a Coprocessgetline
getline
Variantsprint
Statementprint
Statement Examplesprint
printf
Statements for Fancier Printing
printf
Statementprintf
Formatsprintf
print
and printf
BEGIN
and END
Special Patterns
BEGIN
andEND
RulesBEGINFILE
and ENDFILE
Special Patternsif
-else
Statementwhile
Statementdo
-while
Statementfor
Statementswitch
Statementbreak
Statementcontinue
Statementnext
Statementnextfile
Statementexit
StatementARGC
and ARGV
delete
Statementsub()
, gsub()
, and gensub()
return
Statementgettext
printf
Argumentschdir()
and stat()
chdir()
and stat()
Arnold Robbins and I are good friends. We were introducedin 1990by circumstances—and our favorite programming language, AWK. The circumstances started a couple of yearsearlier. I was working at a new job and noticed an unpluggedUnix computer sitting in the corner. No one knew how to use it,and neither did I. However,a couple of days later it was running, andI wasroot
and the one-and-only user. That day, I began the transition from statistician to Unix programmer.
On one of many trips to the library or bookstore in search ofbooks on Unix, I found the gray AWK book, a.k.a. Aho, Kernighan andWeinberger,The AWK Programming Language, Addison-Wesley,1988. AWK's simple programming paradigm—find a pattern in theinput and then perform an action—often reduced complex or tediousdata manipulations to few lines of code. I was excited to try myhand at programming in AWK.
Alas, the awk on my computer was a limited version of thelanguage described in the AWK book. I discovered that my computerhad “oldawk” and the AWK book described “new awk.”I learned that this was typical; the old version refused to stepaside or relinquish its name. If a system had a newawk, it wasinvariably called nawk, and few systems had it. The best way to get a newawk was to ftp the source code forgawk fromprep.ai.mit.edu
. gawk was a version ofnewawk written by David Trueman and Arnold, and available underthe GNU General Public License.
(Incidentally,it's no longer difficult to find a new awk.gawk ships withGNU/Linux, and you can download binaries or source code for almostany system; my wife usesgawk on her VMS box.)
My Unix system started out unplugged from the wall; it certainly was notplugged into a network. So, oblivious to the existence ofgawkand the Unix community in general, and desiring a newawk, I wrotemy own, called mawk. Before I was finished I knew aboutgawk,but it was too late to stop, so I eventually postedto acomp.sources
newsgroup.
A few days after my posting, I got a friendly emailfrom Arnold introducinghimself. He suggested we share design and algorithms andattached a draft of the POSIX standard sothat I could updatemawk to support language extensions addedafter publication of the AWK book.
Frankly, if our roles hadbeen reversed, I would not have been so open and we probably wouldhave never met. I'm glad we did meet. He is an AWK expert's AWK expert and a genuinely nice person. Arnold contributes significant amounts of hisexpertise and time to the Free Software Foundation.
This book is the gawk reference manual, but at its core itis a book about AWK programming thatwill appeal to a wide audience. It is a definitive reference to the AWK language as defined by the1987 Bell Laboratories release and codified in the 1992 POSIX Utilitiesstandard.
On the other hand, the novice AWK programmer can studya wealth of practical programs that emphasizethe power of AWK's basic idioms:data driven control-flow, pattern matching with regular expressions,and associative arrays. Those looking for something new can try out gawk'sinterface to network protocols via special/inet files.
The programs in this book make clear that an AWK program istypically much smaller and faster to develop thana counterpart written in C. Consequently, there is often a payoff to prototype analgorithm or design in AWK to get it running quickly and exposeproblems early. Often, the interpreted performance is adequateand the AWK prototype becomes the product.
The new pgawk (profiling gawk), producesprogram execution counts. I recently experimented with an algorithm that forn lines of input, exhibited~ C n^2performance, whiletheory predicted~ C n log nbehavior. A few minutes poringover the awkprof.out profile pinpointed the problem toa single line of code.pgawk is a welcome addition tomy programmer's toolbox.
Arnold has distilled over a decade of experience writing andusing AWK programs, and developinggawk, into this book. If you useAWK or want to learn how, then read this book.
Michael Brennan
Author of mawk
March, 2001
Several kinds of tasks occur repeatedlywhen working with text files. You might want to extract certain lines and discard the rest. Or you may need to make changes wherever certain patterns appear,but leave the rest of the file alone. Writing single-use programs for these tasks in languages such as C, C++,or Java is time-consuming and inconvenient. Such jobs are often easier withawk. The awk utility interprets a special-purpose programming languagethat makes it easy to handle simple data-reformatting jobs.
The GNU implementation of awk is calledgawk; it is fullycompatible withthe POSIX1specification of theawk languageand with the Unix version ofawk maintainedby Brian Kernighan. This means that allproperly writtenawk programs should work with gawk. Thus, we usually don't distinguish betweengawk and otherawk implementations.
Usingawk allows you to:
In addition,gawkprovides facilities that make it easy to:
This Web page teaches you about the awk language andhow you can use it effectively. You should already be familiar with basicsystem commands, such ascat and ls,2 as well as basic shellfacilities, such as input/output (I/O) redirection and pipes.
Implementations of theawk language are available for manydifferent computing environments. This Web page, while describingtheawk language in general, also describes the particularimplementation ofawk called gawk (which stands for“GNU awk”).gawk runs on a broad range of Unix systems,ranging from Intel®-architecture PC-based computersup through large-scale systems,such as Crays.gawk has also been ported to Mac OS X,Microsoft Windows (all versions) and OS/2 PCs,and VMS. (Some other, obsolete systems to whichgawk was once portedare no longer supported and the code for those systemshas been removed.)
1 part egrep |
1 part snobol |
|
2 parts ed |
3 parts C |
Blend all parts well usinglex
andyacc
. Document minimally and release.After eight years, add another part
egrep
and twomore parts C. Document very well and release.
The nameawk comes from the initials of its designers: Alfred V. Aho, Peter J. Weinberger and Brian W. Kernighan. The original version ofawk was written in 1977 at AT&T Bell Laboratories. In 1985, a new version made the programminglanguage more powerful, introducing user-defined functions, multiple inputstreams, and computed regular expressions. This new version became widely available with Unix System VRelease 3.1 (1987). The version in System V Release 4 (1989) added some new features and cleanedup the behavior in some of the “dark corners” of the language. The specification forawk in the POSIX Command Languageand Utilities standard further clarified the language. Both thegawk designers and the original Bell Laboratoriesawkdesigners provided feedback for the POSIX specification.
Paul Rubin wrote the GNU implementation,gawk, in 1986. Jay Fenlason completed it, with advice from Richard Stallman. John Woodscontributed parts of the code as well. In 1988 and 1989, David Trueman, withhelp from me, thoroughly reworkedgawk for compatibilitywith the newer awk. Circa 1994, I became the primary maintainer. Current development focuses on bug fixes,performance improvements, standards compliance, and occasionally, new features.
In May of 1997, Jürgen Kahrs felt the need for network accessfrom awk, and with a little help from me, set about addingfeatures to do this forgawk. At that time, he alsowrote the bulk ofTCP/IP Internetworking withgawk(a separate document, available as part of thegawk distribution). His code finally became part of the maingawk distributionwith gawk version 3.1.
John Haque rewrote the gawk internals, in the process providinganawk-level debugger. This version became available asgawk version 4.0, in 2011.
See Contributors,for a complete list of those who made important contributions togawk.
Theawk language has evolved over the years. Full details areprovided inLanguage History. The language described in this Web pageis often referred to as “newawk” (nawk).
Because of this, there are systems with multipleversions ofawk. Some systems have an awk utility that implements theoriginal version of theawk language and a nawk utilityfor the new version. Others have anoawk version for the “old awk”language and plainawk for the new one. Still others onlyhave one version, which is usually the new one.3
All in all, this makes it difficult for you to know which version ofawk you should run when writing your programs. The best advicewe can give here is to check your local documentation. Look forawk,oawk, andnawk, as well as for gawk. It is likely that you alreadyhave some version of newawk on your system, which is whatyou should use when running your programs. (Of course, if you're readingthis Web page, chances are good that you havegawk!)
Throughout this Web page, whenever we refer to a language featurethat should be available in any complete implementation of POSIXawk,we simply use the term awk. When referring to a feature that isspecific to the GNU implementation, we use the termgawk.
The termawk refers to a particular program as well as to the language youuse to tell this program what to do. When we need to be careful, we callthe language “theawk language,”and the program “the awk utility.”This Web page explainsboth how to write programs in theawk language and how torun the awk utility. The termawk program refers to a program written by you intheawk programming language.
Primarily, this Web page explains the features of awkas defined in the POSIX standard. It does so in the context of thegawk implementation. While doing so, it alsoattempts to describe important differences between gawkand other awk implementations.4Finally, anygawk features that are not inthe POSIX standard forawk are noted.
This Web page has the difficult task of being both a tutorial and a reference. If you are a novice, feel free to skip over details that seem too complex. You should also ignore the many cross-references; they are for theexpert user and for the online Info and HTML versions of the document.
There aresubsections labeledas Advanced Notesscattered throughout the Web page. They add a more complete explanation of points that are relevant, but not likelyto be of interest on first reading. All appear in the index, under the heading “advanced features.”
Most of the time, the examples use complete awk programs. Some of the more advanced sections show only the part of theawkprogram that illustrates the concept currently being described.
While this Web page is aimed principally at people who have not beenexposedto awk, there is a lot of information here that even theawkexpert should find useful. In particular, the description of POSIXawk and the example programs inLibrary Functions, and inSample Programs,should be of interest.
Getting Started,provides the essentials you need to know to begin usingawk.
Invoking Gawk,describes how to run gawk, the meaning of itscommand-line options, and how it findsawkprogram source files.
Regexp,introduces regular expressions in general, and in particular the flavorssupported by POSIXawk and gawk.
Reading Files,describes how awk reads your data. It introduces the concepts of records and fields, as wellas thegetline
command. I/O redirection is first described here. Network I/O is also briefly introduced here.
Printing,describes how awk programs can produce output withprint
andprintf
.
Expressions,describes expressions, which are the basic building blocksfor getting most things done in a program.
Patterns and Actions,describes how to write patterns for matching records, actions fordoing something when a record is matched, and the built-in variablesawk andgawk use.
Arrays,covers awk's one-and-only data structure: associative arrays. Deleting array elements and whole arrays is also described, as well assorting arrays ingawk. It also describes how gawkprovides arrays of arrays.
Functions,describes the built-in functions awk andgawk provide, as well as how to defineyour own functions.
Internationalization,describes special features ingawk for translating programmessages into different languages at runtime.
Advanced Features,describes a number of gawk-specific advanced features. Of particular noteare the abilities to have two-way communications with another process,perform TCP/IP networking, andprofile yourawk programs.
Library Functions, andSample Programs,provide many sampleawk programs. Reading them allows you to seeawksolving real problems.
Debugger, describes the awk debugger,dgawk.
Language History,describes how the awk language has evolved sinceits first release to present. It also describes howgawkhas acquired features over time.
Installation,describes how to get gawk, how to compile iton POSIX-compatible systems,and how to compile and use it on differentnon-POSIX systems. It also describes how to report bugsingawk and where to get other freelyavailableawk implementations.
Notes,describes how to disable gawk's extensions, aswell as how to contribute new code togawk,how to write extension libraries, and some possiblefuture directions forgawk development.
Basic Concepts,provides some very cursory background material for those whoare completely unfamiliar with computer programming. Also centralized there is a discussion of some of the issuessurrounding floating-point numbers.
TheGlossary,defines most, if not all, the significant terms usedthroughout the book. If you find terms that you aren't familiar with, try looking them up here.
Copying, andGNU Free Documentation License,present the licenses that cover thegawk source codeand this Web page, respectively.
This Web page is written in Texinfo,the GNU documentation formatting language. A single Texinfo source file is used to produce both the printed and onlineversions of the documentation. Because of this, the typographical conventionsare slightly different than in other books you may have read.
Examples you would type at the command-line are preceded by the commonshell primary and secondary prompts, ‘$’ and ‘>’. Input that you type is shownlike this. Output from the command is preceded by the glyph “-|”. This typically represents the command's standard output. Error messages, and other output on the command's standard error, are precededby the glyph “error-->”. For example:
$ echo hi on stdout -| hi on stdout $ echo hello on stderr 1>&2 error--> hello on stderr
In the text, command names appear in this font
, while code segmentsappear in the same font and quoted, ‘like this’. Options look like this:-f. Some things areemphasized like this, and if a point needs to be madestrongly, it is donelike this. The first occurrence ofa new term is usually its definition and appears in the samefont as the previous occurrence of “definition” in this sentence. Finally, file names are indicated like this:/path/to/ourfile.
Characters that you type at the keyboard look like this. In particular,there are special characters called “control characters.” These arecharacters that you type by holding down both theCONTROL key andanother key, at the same time. For example, a Ctrl-d is typedby first pressing and holding theCONTROL key, nextpressing the d key and finally releasing both keys.
Dark corners are basically fractal — no matter how muchyou illuminate, there's always a smaller but darker one.
Brian Kernighan
Until the POSIX standard (andGAWK: Effective AWK Programming),many features of awk were either poorly documented or notdocumented at all. Descriptions of such features(often called “dark corners”) are noted in this Web page with“(d.c.)”. They also appear in the index under the heading “dark corner.”
As noted by the opening quote, though, anycoverage of dark cornersis, by definition, incomplete.
Extensions to the standard awk language that are supported bymore than oneawk implementation are marked“(c.e.),” and listed in the index under “common extensions”and “extensions, common.”
The Free Software Foundation (FSF) is a nonprofit organization dedicatedto the production and distribution of freely distributable software. It was founded by Richard M. Stallman, the author of the originalEmacs editor. GNU Emacs is the most widely used version of Emacs today.
The GNU5Project is an ongoing effort on the part of the Free SoftwareFoundation to create a complete, freely distributable, POSIX-compliantcomputing environment. The FSF uses the “GNU General Public License” (GPL) to ensure thattheir software'ssource code is always available to the end user. Acopy of the GPL is includedin this Web pagefor your reference(seeCopying). The GPL applies to the C language source code forgawk. To find out more about the FSF and the GNU Project online,seethe GNU Project's home page. This Web page may also be read fromtheir web site.
A shell, an editor (Emacs), highly portable optimizing C, C++, andObjective-C compilers, a symbolic debugger and dozens of large andsmall utilities (such asgawk), have all been completed and arefreely available. The GNU operatingsystem kernel (the HURD), has been released but remains in an earlystage of development.
Until the GNU operating system is more fully developed, you shouldconsider using GNU/Linux, a freely distributable, Unix-like operatingsystem for Intel®,Power Architecture,Sun SPARC, IBM S/390, and othersystems.6Many GNU/Linux distributions areavailable for download from the Internet.
(There are numerous other freely available, Unix-like operating systemsbased on theBerkeley Software Distribution, and some of them use recent versionsofgawk for their versions of awk.NetBSD,FreeBSD,andOpenBSDare three of the most popular ones, but thereare others.)
The Web page you are reading is actually free—at least, theinformation in it is free to anyone. The machine-readablesource code for the Web page comes withgawk; anyonemay take this Web page to a copying machine and make as manycopies as they like. (Take a moment to check the Free DocumentationLicense inGNU Free Documentation License.)
The Web page itself has gone through a number of previous editions. Paul Rubin wrote the very first draft ofThe GAWK Manual;it was around 40 pages in size. Diane Close and Richard Stallman improved it, yielding aversion that wasaround 90 pages long and barely described the original, “old”version ofawk.
I started working with that version in the fall of 1988. As work on it progressed,the FSF published several preliminary versions (numbered 0.x). In 1996, Edition 1.0 was released withgawk 3.0.0. The FSF published the first two editions underthe titleThe GNU Awk User's Guide.
This edition maintains the basic structure of the previous editions. For Edition 4.0, the content has been thoroughly reviewedand updated. All references to versions prior to 4.0 have beenremoved. Of significant note for this edition isDebugger.
GAWK: Effective AWK Programming will undoubtedly continue to evolve. An electronic versioncomes with thegawk distribution from the FSF. If you find an error in this Web page, please report it! SeeBugs, for information on submittingproblem reports electronically.
As the maintainer of GNU awk, I once thought that I would beable to manage a collection of publicly availableawk programsand I even solicited contributions. Making things available on the Internethelps keep thegawk distribution down to manageable size.
The initial collection of material, such as it is, is still availableat ftp://ftp.freefriends.org/arnold/Awkstuff. In the hopes ofdoing something more broad, I acquired theawk.info
domain.
However, I found that I could not dedicate enough time to managingcontributed code: the archive did not grow and the domain went unusedfor several years.
Fortunately, late in 2008, a volunteer took on the task of setting upan awk-related web site—http://awk.info—and did a verynice job.
If you have written an interesting awk program, or have writtenagawk extension that you would like to share with the restof the world, please seehttp://awk.info/?contribute for how tocontribute it to the web site.
The initial draft of The GAWK Manual had the following acknowledgments:
Many people need to be thanked for their assistance in producing thismanual. Jay Fenlason contributed many ideas and sample programs. RichardMlynarik and Robert Chassell gave helpful comments on drafts of thismanual. The paper A Supplemental Document for awk by John W. Pierce of the Chemistry Department at UC San Diego, pinpointed severalissues relevant both to awk implementation and to this manual, thatwould otherwise have escaped us.
I would like to acknowledge Richard M. Stallman, for his vision of abetter world and for his courage in founding the FSF and starting theGNU Project.
Earlier editions of this Web page had the following acknowledgements:
The following people (in alphabetical order)provided helpful comments on variousversions of this book,Rick Adams,Dr. Nelson H.F. Beebe,Karl Berry,Dr. Michael Brennan,Rich Burridge,Claire Cloutier,Diane Close,Scott Deifik,Christopher (“Topher”) Eliot,Jeffrey Friedl,Dr. Darrel Hankerson,Michal Jaegermann,Dr. Richard J. LeBlanc,Michael Lijewski,Pat Rankin,Miriam Robbins,Mary Sheehan,andChuck Toporek.Robert J. Chassell provided much valuable advice onthe use of Texinfo. He also deserves special thanks forconvincing menot to title this Web pageHow To Gawk Politely. Karl Berry helped significantly with the TeX part of Texinfo.
I would like to thank Marshall and Elaine Hartholz of Seattle andDr. Bert and Rita Schreiber of Detroit for large amounts of quiet vacationtime in their homes, which allowed me to make significant progress onthis Web page and ongawk itself.
Phil Hughes of SSCcontributed in a very important way by loaning me his laptop GNU/Linuxsystem, not once, but twice, which allowed me to do a lot of work whileaway from home.
David Trueman deserves special credit; he has done a yeoman jobof evolvinggawk so that it performs well and without bugs. Although he is no longer involved withgawk,working with him on this project was a significant pleasure.
The intrepid members of the GNITS mailing list, and most notably UlrichDrepper, provided invaluable help and feedback for the design of theinternationalization features.
Chuck Toporek, Mary Sheehan, and Claire Coutier of O'Reilly & Associates contributedsignificant editorial help for this Web page for the3.1 release ofgawk.
Dr. Nelson Beebe,Andreas Buening,Antonio Colombo,Stephen Davies,Scott Deifik,John H. DuBois III,Darrel Hankerson,Michal Jaegermann,Jürgen Kahrs,Dave Pitts,Stepan Kasal,Pat Rankin,Andrew Schorr,Corinna Vinschen,Anders Wallin,and Eli Zaretskii(in alphabetical order)make up the currentgawk “crack portability team.” Without their hard work andhelp,gawk would not be nearly the fine program it is today. Ithas been and continues to be a pleasure working with this team of finepeople.
John Haque contributed the modifications to convert gawkinto a byte-code interpreter, including the debugger. Stephen Daviescontributed to the effort to bring the byte-code changes into the mainstreamcode base. Efraim Yawitz contributed the initial text of Debugger.
I would like to thank Brian Kernighan for invaluable assistance during thetesting and debugging ofgawk, and for ongoinghelp and advice in clarifying numerous points about the language. We could not have done nearly as good a job on eithergawkor its documentation without his help.
I must thank my wonderful wife, Miriam, for her patience throughthe many versions of this project, for her proofreading,and for sharing me with the computer. I would like to thank my parents for their love, and for the grace withwhich they raised and educated me. Finally, I also must acknowledge my gratitude to G-d, for the many opportunitiesHe has sent my way, as well as for the gifts He has given me with which totake advantage of those opportunities.
Arnold Robbins
The basic function ofawk is to search files for lines (or otherunits of text) that contain certain patterns. When a line matches oneof the patterns,awk performs specified actions on that line.awk keeps processing input lines in this way until it reachesthe end of the input files.
Programs inawk are different from programs in most other languages,becauseawk programs are data-driven; that is, you describethe data you want to work with and then what to do when you find it. Most other languages areprocedural; you have to describe, in greatdetail, every step the program is to take. When working with procedurallanguages, it is usually muchharder to clearly describe the data your program will process. For this reason,awk programs are often refreshingly easy toread and write.
When you runawk, you specify an awkprogram thattells awk what to do. The program consists of a series ofrules. (It may also containfunction definitions,an advanced feature that we will ignore for now. SeeUser-defined.) Each rule specifies onepattern to search for and one action to performupon finding the pattern.
Syntactically, a rule consists of a pattern followed by an action. Theaction is enclosed in curly braces to separate it from the pattern. Newlines usually separate rules. Therefore, anawkprogram looks like this:
pattern { action } pattern { action } ...
There are several ways to run anawk program. If the program isshort, it is easiest to include it in the command that runsawk,like this:
awk 'program' input-file1 input-file2 ...
When the program is long, it is usually more convenient to put it in a fileand run it with a command like this:
awk -f program-file input-file1 input-file2 ...
This section discusses both mechanisms, along with severalvariations of each.
Once you are familiar with awk, you will often type in simpleprograms the moment you want to use them. Then you can write theprogram as the first argument of theawk command, like this:
awk 'program' input-file1 input-file2 ...
where program consists of a series of patterns andactions, as described earlier.
This command format instructs theshell, or command interpreter,to start awk and use theprogram to process records in theinput file(s). There are single quotes aroundprogram sothe shell won't interpret any awk characters as special shellcharacters. The quotes also cause the shell to treat all ofprogram asa single argument for awk, and allowprogram to be morethan one line long.
This format is also useful for running short or medium-sizedawkprograms from shell scripts, because it avoids the need for a separatefile for theawk program. A self-contained shell script is morereliable because there are no other files to misplace.
Very Simple,later in this chapter,presents several short,self-contained programs.
You can also runawk without any input files. If you type thefollowing command line:
awk 'program'
awk applies the program to the standard input,which usually means whatever you type on the terminal. This continuesuntil you indicate end-of-file by typingCtrl-d. (On other operating systems, the end-of-file character may be different. For example, on OS/2, it isCtrl-z.)
As an example, the following program prints a friendly piece of advice(from Douglas Adams's The Hitchhiker's Guide to the Galaxy),to keep you from worrying about the complexities of computerprogramming7(BEGIN
is a feature we haven't discussed yet):
$ awk "BEGIN { print \"Don't Panic!\" }" -| Don't Panic!
This program does not read any input. The ‘\’ before each of theinner double quotes is necessary because of the shell's quotingrules—in particular because it mixes both single quotes anddouble quotes.8
This next simple awk programemulates thecat utility; it copies whatever you type on thekeyboard to its standard output (why this works is explained shortly).
$ awk '{ print }' Now is the time for all good men -| Now is the time for all good men to come to the aid of their country. -| to come to the aid of their country. Four score and seven years ago, ... -| Four score and seven years ago, ... What, me worry? -| What, me worry? Ctrl-d
Sometimes yourawk programs can be very long. In this case, it ismore convenient to put the program into a separate file. In order to tellawk to use that file for its program, you type:
awk -f source-file input-file1 input-file2 ...
The-f instructs the awk utility to get theawk programfrom the file source-file. Any file name can be used forsource-file. For example, you could put the program:
BEGIN { print "Don't Panic!" }
into the file advice. Then this command:
awk -f advice
does the same thing as this one:
awk "BEGIN { print \"Don't Panic!\" }"
This was explained earlier(see Read Terminal). Note that you don't usually need single quotes around the file name that youspecify with-f, because most file names don't contain any of the shell'sspecial characters. Notice that inadvice, the awkprogram did not have single quotes around it. The quotes are only neededfor programs that are provided on theawk command line.
If you want to clearly identify yourawk program files as such,you can add the extension.awk to the file name. This doesn'taffect the execution of theawk program but it does make“housekeeping” easier.
Once you have learned awk, you may want to write self-containedawk scripts, using the ‘#!’ script mechanism. You can dothis on many systems.9For example, you could update the file advice to look like this:
#! /bin/awk -f BEGIN { print "Don't Panic!" }
After making this file executable (with the chmod utility),simply type ‘advice’at the shell and the system arranges to runawk10 as if you hadtyped ‘awk -f advice’:
$ chmod +x advice $ advice -| Don't Panic!
(We assume you have the current directory in your shell's searchpath variable [typically$PATH
]. If not, you may needto type ‘./advice’ at the shell.)
Self-contained awk scripts are useful when you want to write aprogram that users can invoke without their having to know that the program iswritten inawk.
Some systems limit the length of the interpreter name to 32 characters. Often, this can be dealt with by using a symbolic link.
You should not put more than one argument on the ‘#!’line after the path toawk. It does not work. The operating systemtreats the rest of the line as a single argument and passes it toawk. Doing this leads to confusing behavior—most likely a usage diagnosticof some sort fromawk.
Finally,the value ofARGV[0]
(see Built-in Variables)varies depending upon your operating system. Some systems put ‘awk’ there, some put the full pathnameofawk (such as /bin/awk), and some put the nameof your script (‘advice’). (d.c.) Don't rely on the value ofARGV[0]
to provide your script name.
Acomment is some text that is included in a program for the sakeof human readers; it is not really an executable part of the program. Commentscan explain what the program does and how it works. Nearly allprogramming languages have provisions for comments, as programs aretypically hard to understand without them.
In the awk language, a comment starts with the sharp signcharacter (‘#’) and continues to the end of the line. The ‘#’ does not have to be the first character on the line. Theawk language ignores the rest of a line following a sharp sign. For example, we could have put the following intoadvice:
# This program prints a nice friendly message. It helps # keep novice users from being afraid of the computer. BEGIN { print "Don't Panic!" }
You can put comment lines into keyboard-composed throwaway awkprograms, but this usually isn't very useful; the purpose of acomment is to help you or another person understand the programwhen reading it at a later time.
CAUTION: As mentioned in One-shot,you can enclose small to medium programs in single quotes, in order to keepyour shell scripts self-contained. When doing so, don't putan apostrophe (i.e., a single quote) into a comment (or anywhere elsein your program). The shell interprets the quote as the closingquote for the entire program. As a result, usually the shellprints a message about mismatched quotes, and if awk actuallyruns, it will probably print strange messages about syntax errors. For example, look at the following:$ awk '{ print "hello" } # let's be cute' >The shell sees that the first two quotes match, and thata new quoted object begins at the end of the command line. It therefore prompts with the secondary prompt, waiting for more input. With Unixawk, closing the quoted string produces this result:
$ awk '{ print "hello" } # let's be cute' > ' error--> awk: can't open file be error--> source line number 1Putting a backslash before the single quote in ‘let's’ wouldn't help,since backslashes are not special inside single quotes. The next subsection describes the shell's quoting rules.
For short to medium length awk programs, it is most convenientto enter the program on theawk command line. This is best done by enclosing the entire program in single quotes. This is true whether you are entering the program interactively atthe shell prompt, or writing it as part of a larger shell script:
awk 'program text' input-file1 input-file2 ...
Once you are working with the shell, it is helpful to have a basicknowledge of shell quoting rules. The following rules apply only toPOSIX-compliant, Bourne-style shells (such as Bash, the GNU Bourne-AgainShell). If you use the C shell, you're on your own.
Since certain characters within double-quoted text are processed by the shell,they must beescaped within the text. Of note are the characters‘$’, ‘`’, ‘\’, and ‘"’, all of which must be preceded bya backslash within double-quoted text if they are to be passed on literallyto the program. (The leading backslash is stripped first.) Thus, the example seenpreviouslyinRead Terminal,is applicable:
$ awk "BEGIN { print \"Don't Panic!\" }" -| Don't Panic!
Note that the single quote is not special within double quotes.
FS
shouldbe set to the null string, use: awk -F "" 'program' files # correct
Don't use this:
awk -F"" 'program' files # wrong!
In the second case, awk will attempt to use the text of the programas the value ofFS
, and the first file name as the text of the program! This results in syntax errors at best, and confusing behavior at worst.
Mixing single and double quotes is difficult. You have to resortto shell quoting tricks, like this:
$ awk 'BEGIN { print "Here is a single quote <'"'"'>" }' -| Here is a single quote <'>
This program consists of three concatenated quoted strings. The first and thethird are single-quoted, the second is double-quoted.
This can be “simplified” to:
$ awk 'BEGIN { print "Here is a single quote <'\''>" }' -| Here is a single quote <'>
Judge for yourself which of these two is the more readable.
Another option is to use double quotes, escaping the embedded, awk-leveldouble quotes:
$ awk "BEGIN { print \"Here is a single quote <'>\" }" -| Here is a single quote <'>
This option is also painful, because double quotes, backslashes, and dollar signsare very common in more advancedawk programs.
A third option is to use the octal escape sequence equivalents(see Escape Sequences)for thesingle- and double-quote characters, like so:
$ awk 'BEGIN { print "Here is a single quote <\47>" }' -| Here is a single quote <'> $ awk 'BEGIN { print "Here is a double quote <\42>" }' -| Here is a double quote <">
This works nicely, except that you should comment clearly what theescapes mean.
A fourth option is to use command-line variable assignment, like this:
$ awk -v sq="'" 'BEGIN { print "Here is a single quote <" sq ">" }' -| Here is a single quote <'>
If you really need both single and double quotes in your awkprogram, it is probably best to move it into a separate file, wherethe shell won't be part of the picture, and you can say what you mean.
Although this Web page generally only worries about POSIX systems and thePOSIX shell, the following issue arises often enough for many users thatit is worth addressing.
The “shells” on Microsoft Windows systems use the double-quotecharacter for quoting, and make it difficult or impossible to include anescaped double-quote character in a command-line script. The following example, courtesy of Jeroen Brink, showshow to print all lines in a file surrounded by double quotes:
gawk "{ print \"\042\" $0 \"\042\" }" file
Many of the examples in this Web page take their input from two sampledata files. The first,BBS-list, represents a list ofcomputer bulletin board systems together with information about those systems. The second data file, calledinventory-shipped, containsinformation about monthly shipments. In both files,each line is considered to be onerecord.
In the data file BBS-list, each record contains the name of a computerbulletin board, its phone number, the board's baud rate(s), and a code forthe number of hours it is operational. An ‘A’ in the last columnmeans the board operates 24 hours a day. A ‘B’ in the lastcolumn means the board only operates on evening and weekend hours. A ‘C’ means the board operates only on weekends:
aardvark 555-5553 1200/300 B alpo-net 555-3412 2400/1200/300 A barfly 555-7685 1200/300 A bites 555-1675 2400/1200/300 A camelot 555-0542 300 C core 555-2912 1200/300 C fooey 555-1234 2400/1200/300 B foot 555-6699 1200/300 B macfoo 555-6480 1200/300 A sdace 555-3430 2400/1200/300 A sabafoo 555-2127 1200/300 C
The data fileinventory-shipped representsinformation about shipments during the year. Each record contains the month, the numberof green crates shipped, the number of red boxes shipped, the number oforange bags shipped, and the number of blue packages shipped,respectively. There are 16 entries, covering the 12 months of last yearand the first four months of the current year.
Jan 13 25 15 115 Feb 15 32 24 226 Mar 15 24 34 228 Apr 31 52 63 420 May 16 34 29 208 Jun 31 42 75 492 Jul 24 34 67 436 Aug 15 34 47 316 Sep 13 55 37 277 Oct 29 54 68 525 Nov 20 87 82 577 Dec 17 35 61 401 Jan 21 36 64 620 Feb 26 58 80 652 Mar 24 75 70 495 Apr 21 70 74 514
The following command runs a simple awk program that searches theinput fileBBS-list for the character string ‘foo’ (agrouping of characters is usually called astring;the term string is based on similar usage in English, suchas “a string of pearls,” or “a string of cars in a train”):
awk '/foo/ { print $0 }' BBS-list
When lines containing ‘foo’ are found, they are printed because‘print $0’ means print the current line. (Just ‘print’ byitself means the same thing, so we could have written thatinstead.)
You will notice that slashes (‘/’) surround the string ‘foo’in theawk program. The slashes indicate that ‘foo’is the pattern to search for. This type of pattern is called aregular expression, which is covered in more detail later(seeRegexp). The pattern is allowed to match parts of words. There aresingle quotes around theawk program so that the shell won'tinterpret any of it as special shell characters.
Here is what this program prints:
$ awk '/foo/ { print $0 }' BBS-list -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sabafoo 555-2127 1200/300 C
In anawk rule, either the pattern or the action can be omitted,but not both. If the pattern is omitted, then the action is performedforevery input line. If the action is omitted, the defaultaction is to print all lines that match the pattern.
Thus, we could leave out the action (theprint
statement and the curlybraces) in the previous example and the result would be the same:awk prints all lines matching the pattern ‘foo’. By comparison,omitting the print
statement but retaining the curly braces makes anempty action that does nothing (i.e., no lines are printed).
Many practicalawk programs are just a line or two. Following is acollection of useful, short programs to get you started. Some of theseprograms contain constructs that haven't been covered yet. (The descriptionof the program will give you a good idea of what is going on, but pleaseread the rest of the Web page to become anawk expert!) Most of the examples use a data file nameddata. This is just aplaceholder; if you use these programs yourself, substituteyour own file names fordata. For future reference, note that there is often more thanone way to do things inawk. At some point, you may wantto look back at these examples and see ifyou can come up with different ways to do the same things shown here:
awk '{ if (length($0) > max) max = length($0) } END { print max }' data
awk 'length($0) > 80' data
The sole rule has a relational expression as its pattern and it has noaction—so the default action, printing the record, is used.
expand data | awk '{ if (x < length()) x = length() } END { print "maximum line length is " x }'
The input is processed by the expand utility to change TABsinto spaces, so the widths compared are actually the right-margin columns.
awk 'NF > 0' data
This is an easy way to delete blank lines from a file (or rather, tocreate a new file similar to the old file but from which the blank lineshave been removed).
awk 'BEGIN { for (i = 1; i <= 7; i++) print int(101 * rand()) }'
ls -l files | awk '{ x += $5 } END { print "total bytes: " x }'
ls -l files | awk '{ x += $5 } END { print "total K-bytes:", x / 1024 }'
awk -F: '{ print $1 }' /etc/passwd | sort
awk 'END { print NR }' data
awk 'NR % 2 == 0' data
If you use the expression ‘NR % 2 == 1’ instead,the program would print the odd-numbered lines.
The awk utility reads the input files one line at atime. For each line,awk tries the patterns of each of the rules. If several patterns match, then several actions are run in the order inwhich they appear in theawk program. If no patterns match, thenno actions are run.
After processing all the rules that match the line (and perhaps there are none),awk reads the next line. (However,seeNext Statement,and also see Nextfile Statement). This continues until the program reaches the end of the file. For example, the followingawk program contains two rules:
/12/ { print $0 } /21/ { print $0 }
The first rule has the string ‘12’ as thepattern and ‘print $0’ as the action. The second rule has thestring ‘21’ as the pattern and also has ‘print $0’ as theaction. Each rule's action is enclosed in its own pair of braces.
This program prints every line that contains the string‘12’or the string ‘21’. If a line contains bothstrings, it is printed twice, once by each rule.
This is what happens if we run this program on our two sample data files,BBS-list andinventory-shipped:
$ awk '/12/ { print $0 } > /21/ { print $0 }' BBS-list inventory-shipped -| aardvark 555-5553 1200/300 B -| alpo-net 555-3412 2400/1200/300 A -| barfly 555-7685 1200/300 A -| bites 555-1675 2400/1200/300 A -| core 555-2912 1200/300 C -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sdace 555-3430 2400/1200/300 A -| sabafoo 555-2127 1200/300 C -| sabafoo 555-2127 1200/300 C -| Jan 21 36 64 620 -| Apr 21 70 74 514
Note how the line beginning with ‘sabafoo’inBBS-list was printed twice, once for each rule.
Now that we've mastered some simple tasks, let's look atwhat typical awkprograms do. This example shows howawk can be used tosummarize, select, and rearrange the output of another utility. It usesfeatures that haven't been covered yet, so don't worry if you don'tunderstand all the details:
LC_ALL=C ls -l | awk '$6 == "Nov" { sum += $5 } END { print sum }'
This command prints the total number of bytes in all the files in thecurrent directory that were last modified in November (of any year). The ‘ls -l’ part of this example is a system command that givesyou a listing of the files in a directory, including each file's size and the datethe file was last modified. Its output looks like this:
-rw-r--r-- 1 arnold user 1933 Nov 7 13:05 Makefile -rw-r--r-- 1 arnold user 10809 Nov 7 13:03 awk.h -rw-r--r-- 1 arnold user 983 Apr 13 12:14 awk.tab.h -rw-r--r-- 1 arnold user 31869 Jun 15 12:20 awkgram.y -rw-r--r-- 1 arnold user 22414 Nov 7 13:03 awk1.c -rw-r--r-- 1 arnold user 37455 Nov 7 13:03 awk2.c -rw-r--r-- 1 arnold user 27511 Dec 9 13:07 awk3.c -rw-r--r-- 1 arnold user 7989 Nov 7 13:03 awk4.c
The first field contains read-write permissions, the second field containsthe number of links to the file, and the third field identifies the owner ofthe file. The fourth field identifies the group of the file. The fifth field contains the size of the file in bytes. Thesixth, seventh, and eighth fields contain the month, day, and time,respectively, that the file was last modified. Finally, the ninth fieldcontains the file name.11
The ‘$6 == "Nov"’ in ourawk program is an expression thattests whether the sixth field of the output from ‘ls -l’matches the string ‘Nov’. Each time a line has the string‘Nov’ for its sixth field, the action ‘sum += $5’ isperformed. This adds the fifth field (the file's size) to the variablesum
. As a result, whenawk has finished reading all theinput lines,sum
is the total of the sizes of the files whoselines matched the pattern. (This works becauseawk variablesare automatically initialized to zero.)
After the last line of output from ls has been processed, theEND
rule executes and prints the value ofsum
. In this example, the value of sum
is 80600.
These more advanced awk techniques are covered in later sections(seeAction Overview). Before you can move on to moreadvancedawk programming, you have to know how awk interpretsyour input and displays your output. By manipulating fields and usingprint
statements, you can produce some very useful andimpressive-looking reports.
Most often, each line in anawk program is a separate statement orseparate rule, like this:
awk '/12/ { print $0 } /21/ { print $0 }' BBS-list inventory-shipped
However,gawk ignores newlines after any of the followingsymbols and keywords:
, { ? : || && do else
A newline at any other point is considered the end of thestatement.12
If you would like to split a single statement into two lines at a pointwhere a newline would terminate it, you can continue it by ending thefirst line with a backslash character (‘\’). The backslash must bethe final character on the line in order to be recognized as a continuationcharacter. A backslash is allowed anywhere in the statement, evenin the middle of a string or regular expression. For example:
awk '/This regular expression is too long, so continue it\ on the next line/ { print $1 }'
We have generally not used backslash continuation in our sample programs.gawk places no limit on thelength of a line, so backslash continuation is never strictly necessary;it just makes programs more readable. For this same reason, as well asfor clarity, we have kept most statements short in the sample programspresented throughout the Web page. Backslash continuation ismost useful when yourawk program is in a separate source fileinstead of entered from the command line. You should also note thatmanyawk implementations are more particular about where youmay use backslash continuation. For example, they may not allow you tosplit a string constant using backslash continuation. Thus, for maximumportability of yourawk programs, it is best not to split yourlines in the middle of a regular expression or a string.
CAUTION: Backslash continuation does not work as describedwith the C shell. It works for awk programs in files andfor one-shot programs, provided you are using a POSIX-compliantshell, such as the Unix Bourne shell or Bash. But the C shell behavesdifferently! There, you must use two backslashes in a row, followed bya newline. Note also that when using the C shell, every newlinein your awk program must be escaped with a backslash. To illustrate:% awk 'BEGIN { \ ? print \\ ? "hello, world" \ ? }' -| hello, worldHere, the ‘%’ and ‘?’ are the C shell's primary and secondaryprompts, analogous to the standard shell's ‘$’ and ‘>’.
Compare the previous example to how it is done with a POSIX-compliant shell:
$ awk 'BEGIN { > print \ > "hello, world" > }' -| hello, world
awk is a line-oriented language. Each rule's action has tobegin on the same line as the pattern. To have the pattern and actionon separate lines, youmust use backslash continuation; thereis no other option.
Another thing to keep in mind is that backslash continuation andcomments do not mix. As soon asawk sees the ‘#’ thatstarts a comment, it ignoreseverything on the rest of theline. For example:
$ gawk 'BEGIN { print "dont panic" # a friendly \ > BEGIN rule > }' error--> gawk: cmd. line:2: BEGIN rule error--> gawk: cmd. line:2: ^ parse error
In this case, it looks like the backslash would continue the comment onto thenext line. However, the backslash-newline combination is never evennoticed because it is “hidden” inside the comment. Thus, theBEGIN
is noted as a syntax error.
Whenawk statements within one rule are short, you might want to putmore than one of them on a line. This is accomplished by separating the statementswith a semicolon (‘;’). This also applies to the rules themselves. Thus, the program shown at the start of this sectioncould also be written this way:
/12/ { print $0 } ; /21/ { print $0 }
NOTE: The requirement that states that rules on the same line must beseparated with a semicolon was not in the original awklanguage; it was added for consistency with the treatment of statementswithin an action.
The awk language provides a number of predefined, orbuilt-in, variables that your programs can use to get informationfromawk. There are other variables your program can setas well to control howawk processes your data.
In addition, awk provides a number of built-in functions for doingcommon computational and string-related operations.gawk provides built-in functions for working with timestamps,performing bit manipulation, for runtime string translation (internationalization),determining the type of a variable,and array sorting.
As we develop our presentation of the awk language, we introducemost of the variables and many of the functions. They are describedsystematically inBuilt-in Variables, andBuilt-in.
Now that you've seen some of whatawk can do,you might wonder how awk could be useful for you. By usingutility programs, advanced patterns, field separators, arithmeticstatements, and other selection criteria, you can produce much morecomplex output. The awk language is very useful for producingreports from large amounts of raw data, such as summarizing informationfrom the output of other utility programs likels. (See More Complex.)
Programs written with awk are usually much smaller than they wouldbe in other languages. This makesawk programs easy to compose anduse. Often,awk programs can be quickly composed at your keyboard,used once, and thrown away. Becauseawk programs are interpreted, youcan avoid the (usually lengthy) compilation part of the typicaledit-compile-test-debug cycle of software development.
Complex programs have been written in awk, including a completeretargetable assembler for eight-bit microprocessors (seeGlossary, formore information), and a microcode assembler for a special-purpose Prologcomputer. While the originalawk's capabilities were strained by tasksof such complexity, modern versions are more capable. Even Brian Kernighan'sversion ofawk has fewer predefined limits, and thosethat it has are much larger than they used to be.
If you find yourself writingawk scripts of more than, say, a fewhundred lines, you might consider using a different programminglanguage. Emacs Lisp is a good choice if you need sophisticated stringor pattern matching capabilities. The shell is also good at string andpattern matching; in addition, it allows powerful use of the systemutilities. More conventional languages, such as C, C++, and Java, offerbetter facilities for system programming and for managing the complexityof large programs. Programs in these languages may require more linesof source code than the equivalent awk programs, but they areeasier to maintain and usually run more efficiently.
This chapter covers how to run awk, both POSIX-standardand gawk-specific command-line options, and whatawk andgawk do with non-option arguments. It then proceeds to cover how gawk searches for source files,reading standard input along with other files,gawk'senvironment variables, gawk's exit status, using include files,and obsolete and undocumented options and/or features.
Many of the options and features described here are discussed inmore detail later in the Web page; feel free to skip overthings in this chapter that don't interest you right now.
There are two ways to run awk—with an explicit program or withone or more program files. Here are templates for both of them; itemsenclosed in [...] in these templates are optional:
awk [options] -f progfile [--
] file ... awk [options] [--
] 'program' file ...
Besides traditional one-letter POSIX-style options,gawk alsosupports GNU long options.
It is possible to invokeawk with an empty program:
awk '' datafile1 datafile2
Doing so makes little sense, though;awk exitssilently when given an empty program. (d.c.) If--lint hasbeen specified on the command line,gawk issues awarning that the program is empty.
Options begin with a dash and consist of a single character. GNU-style long options consist of two dashes and a keyword. The keyword can be abbreviated, as long as the abbreviation allows the optionto be uniquely identified. If the option takes an argument, then thekeyword is either immediately followed by an equals sign (‘=’) and theargument's value, or the keyword and the argument's value are separatedby whitespace. If a particular option with a value is given more than once, it is thelast value that counts.
Each long option forgawk has a correspondingPOSIX-style short option. The long and short options areinterchangeable in all contexts. The following list describes options mandated by the POSIX standard:
-F
fs
--field-separator
fs
FS
variable to fs(see Field Separators).
-f
source-file
--file
source-file
-v
var
=
val
--assign
var
=
val
BEGIN
rule(see Other Arguments).
The -v option can only set one variable, but it can be usedmore than once, setting another variable each time, like this:‘awk -v foo=1 -v bar=2 ...’.
CAUTION: Using -v to set the values of the built-invariables may lead to surprising results. awk will reset thevalues of those variables as it needs to, possibly ignoring anypredefined value you may have given.
-W
gawk-opt
--
This is useful if you have file names that start with ‘-’,or in shell scripts, if you have file names that will be specifiedby the user that could start with ‘-’. It is also useful for passing options on to theawkprogram; see Getopt Function.
The following list describes gawk-specific options:
-b
--characters-as-bytes
-c
--traditional
-C
--copyright
-d
[ file
]
--dump-variables
[
=
file
]
Having a list of all global variables is a good way to look fortypographical errors in your programs. You would also use this option if you have a large program with a lot offunctions, and you want to be sure that your functions don'tinadvertently use global variables that you meant to be local. (This is a particularly easy mistake to make with simple variablenames likei
, j
, etc.)
-e
program-text
--source
program-text
-E
file
--exec
file
This option is particularly necessary for World Wide Web CGI applicationsthat pass arguments through the URL; using this option prevents a malicious(or other) user from passing in options, assignments, orawk sourcecode (via --source) to the CGI application. This option should be usedwith ‘#!’ scripts (seeExecutable Scripts), like so:
#! /usr/local/bin/gawk -E awk program here ...
-g
--gen-pot
gettext
Portable Object Template file on standardoutput for all string constants that have been marked for translation. See Internationalization,for information about this option.
-h
--help
-L
[
value
]
--lint
[
=value
]
Some warnings are only printed once, even if the dubious constructs theywarn about occur multiple times in yourawk program. Thus,when eliminating problems pointed out by--lint, you should takecare to search for all occurrences of each inappropriate construct. Asawk programs are usually short, doing so is not burdensome.
-n
--non-decimal-data
CAUTION: This option can severely break old programs. Use with care.
-N
--use-lc-numeric
-O
--optimize
-p
[ file
]
--profile
[
=
file
]
When run with gawk, the profile is just a “pretty printed” versionof the program. When run withpgawk, the profile contains executioncounts for each statement in the program in the left margin, and functioncall counts for each function.
-P
--posix
FS
isequal to a single space(seeFields). FS
to be a single TAB character(see Field Separators). If you supply both --traditional and --posix on thecommand line, --posix takes precedence.gawkalso issues a warning if both options are supplied.
-r
--re-interval
-R
file
--command=
file
-S
--sandbox
system()
function,input redirections with
getline
,output redirections with
print
and
printf
,and dynamic extensions. This is particularly useful when you want to run
awk scriptsfrom questionable sources and need to make sure the scriptscan't access your system (other than the specified input data file).
-t
--lint-old
-V
--version
As long as program text has been supplied,any other options are flagged as invalid with a warning message butare otherwise ignored.
In compatibility mode, as a special case, if the value offs suppliedto the -F option is ‘t’, thenFS
is set to the TABcharacter ("\t"
). This is true only for--traditional and notfor --posix(seeField Separators).
The-f option may be used more than once on the command line. If it is,awk reads its program source from all of the named files, asif they had been concatenated together into one big file. This isuseful for creating libraries ofawk functions. These functionscan be written once and then retrieved from a standard place, insteadof having to be included into each individual program. (As mentioned inDefinition Syntax,function names must be unique.)
With standard awk, library functions can still be used, evenif the program is entered at the terminal,by specifying ‘-f /dev/tty’. After typing your program,typeCtrl-d (the end-of-file character) to terminate it. (You may also use ‘-f -’ to read program source from the standardinput but then you will not be able to also use the standard input as asource of data.)
Because it is clumsy using the standard awk mechanisms to mix sourcefile and command-lineawk programs, gawk provides the--source option. This does not require you to pre-empt the standardinput for your source code; it allows you to easily mix command-lineand library source code(see AWKPATH Variable). The --source option may also be used multiple times on the command line.
If no -f or --source option is specified, thengawkuses the first non-option command-line argument as the text of theprogram source code.
If the environment variablePOSIXLY_CORRECT exists,then gawk behaves in strict POSIX mode, exactly as ifyou had supplied the--posix command-line option. Many GNU programs look for this environment variable to turn onstrict POSIX mode. If--lint is supplied on the command lineandgawk turns on POSIX mode because of POSIXLY_CORRECT,then it issues a warning message indicating that POSIXmode is in effect. You would typically set this variable in your shell's startup file. For a Bourne-compatible shell (such as Bash), you would add theselines to the .profile file in your home directory:
POSIXLY_CORRECT=true export POSIXLY_CORRECT
For a C shell-compatibleshell,13you would add this line to the.login file in your home directory:
setenv POSIXLY_CORRECT true
HavingPOSIXLY_CORRECT set is not recommended for daily use,but it is good for testing the portability of your programs to otherenvironments.
Any additional arguments on the command line are normally treated asinput files to be processed in the order specified. However, anargument that has the form var=
value, assignsthe value value to the variablevar—it does not specify afile at all. (SeeAssignment Options.)
All these arguments are made available to your awk program in theARGV
array (seeBuilt-in Variables). Command-line optionsand the program text (if present) are omitted fromARGV
. All other arguments, including variable assignments, areincluded. As each element ofARGV
is processed, gawksets the variableARGIND
to the index in ARGV
of thecurrent element.
The distinction between file name arguments and variable-assignmentarguments is made whenawk is about to open the next input file. At that point in execution, it checks the file name to see whetherit is really a variable assignment; if so,awk sets the variableinstead of reading a file.
Therefore, the variables actually receive the given values after allpreviously specified files have been read. In particular, the values ofvariables assigned in this fashion arenot available inside aBEGIN
rule(see BEGIN/END),because such rules are run before awk begins scanning the argument list.
The variable values given on the command line are processed for escapesequences (seeEscape Sequences). (d.c.)
In some earlier implementations of awk, when a variable assignmentoccurred before any file names, the assignment would happenbeforethe BEGIN
rule was executed. awk's behavior was thusinconsistent; some command-line assignments were available inside theBEGIN
rule, while others were not. Unfortunately,some applications came to dependupon this “feature.” When awk was changed to be more consistent,the-v option was added to accommodate applications that dependedupon the old behavior.
The variable assignment feature is most useful for assigning to variablessuch asRS
, OFS
, and ORS
, which control input andoutput formats before scanning the data files. It is also useful forcontrolling state if multiple passes are needed over a data file. Forexample:
awk 'pass == 1 { pass 1 stuff } pass == 2 { pass 2 stuff }' pass=1 mydata pass=2 mydata
Given the variable assignment feature, the -F option for settingthe value ofFS
is notstrictly necessary. It remains for historical compatibility.
Often, you may wish to read standard input together with other files. For example, you may wish to read one file, read standard input comingfrom a pipe, and then read another file.
The way to name the standard input, with all versions of awk,is to use a single, standalone minus sign or dash, ‘-’. For example:
some_command | awk -f myprog.awk file1 - file2
Here, awk first readsfile1, then it readsthe output of some_command, and finally it readsfile2.
You may also use "-"
to name standard input when readingfiles withgetline
(see Getline/File).
In addition, gawk allows you to specify the specialfile name/dev/stdin, both on the command line andwithgetline
. Some other versions of awk also support this, but itis not standard. (Some operating systems provide a/dev/stdin filein the file system, however,gawk always processesthis file name itself.)
A number of environment variables influence how gawkbehaves.
In most awkimplementations, you must supply a precise path name for each programfile, unless the file is in the current directory. But ingawk, if the file name supplied to the -f optiondoes not contain a ‘/’, thengawk searches a list ofdirectories (called thesearch path), one by one, looking for afile with the specified name.
The search path is a string consisting of directory namesseparated by colons. gawk gets its search path from theAWKPATH environment variable. If that variable does not exist,gawk uses a default path,‘.:/usr/local/share/awk’.14
The search path feature is particularly useful for building librariesof usefulawk functions. The library files can be placed in astandard directory in the default path and then specified onthe command line with a short file name. Otherwise, the full file namewould have to be typed for each file.
By using both the --source and -f options, your command-lineawk programs can use facilities inawk library files(see Library Functions). Path searching is not done if gawk is in compatibility mode. This is true for both--traditional and --posix. SeeOptions.
NOTE: To includethe current directory in the path, either place . explicitly in the path or write a null entry in thepath. (A null entry is indicated by starting or ending the path with acolon or by placing two colons next to each other (‘ ::’).) This path search mechanism is similarto the shell's.However, gawk always looks in the current directorybeforesearching AWKPATH, so there is no real reason to includethe current directory in the search path.
If AWKPATH is not defined in theenvironment,gawk places its default search path intoENVIRON["AWKPATH"]
. This makes it easy to determinethe actual search path thatgawk will usefrom within an awk program.
While you can change ENVIRON["AWKPATH"]
within your awkprogram, this has no effect on the running program's behavior. This makessense: theAWKPATH environment variable is used to find the programsource files. Once your program is running, all the files have beenfound, andgawk no longer needs to use AWKPATH.
A number of other environment variables affect gawk'sbehavior, but they are more specialized. Those in the followinglist are meant to be used by regular users.
usleep()
system call,the value is rounded up to an integral number of seconds.
The environment variables in the following list are meantfor use by the gawk developers for testing and tuning. They are subject to change. The variables are:
mtrace()
librarycalls from GNU LIBC to help track down possible memory leaks.
If the exit
statement is used with a value(see Exit Statement), thengawk exits withthe numeric value given to it.
Otherwise, if there were no problems during execution,gawk exits with the value of the C constantEXIT_SUCCESS
. This is usually zero.
If an error occurs, gawk exits with the value ofthe C constantEXIT_FAILURE
. This is usually one.
If gawk exits because of a fatal error, the exitstatus is 2. On non-POSIX systems, this value may be mappedtoEXIT_FAILURE
.
This section describes a feature that is specific to gawk.
The ‘@include’ keyword can be used to read externalawk sourcefiles. This gives you the ability to split largeawk source filesinto smaller, more manageable pieces, and also lets you reuse commonawkcode from various awk scripts. In other words, you can grouptogetherawk functions, used to carry out specific tasks,into external files. These files can be used just like function libraries,using the ‘@include’ keyword in conjunction with theAWKPATH
environment variable.
Let's see an example. We'll start with two (trivial) awk scripts, namelytest1 andtest2. Here is the test1 script:
BEGIN { print "This is script test1." }
and here is test2:
@include "test1" BEGIN { print "This is script test2." }
Running gawk with test2produces the following result:
$ gawk -f test2 -| This is file test1. -| This is file test2.
gawk
runs the test2 script which includestest1using the ‘@include’keyword. So, to include externalawk source files you justuse ‘@include’ followed by the name of the file to be included,enclosed in double quotes.
NOTE: Keep in mind that this is a language construct and the file name cannotbe a string variable, but rather just a literal string in double quotes.
The files to be included may be nested; e.g., given a thirdscript, namely test3:
@include "test2" BEGIN { print "This is script test3." }
Running gawk with thetest3 script produces thefollowing results:
$ gawk -f test3 -| This is file test1. -| This is file test2. -| This is file test3.
The file name can, of course, be a pathname. For example:
@include "../io_funcs"
or:
@include "/usr/awklib/network"
are valid. The AWKPATH
environment variable can be of greatvalue when using ‘@include’. The same rules for the useof theAWKPATH
variable in command-line file searches(see AWKPATH Variable) apply to‘@include’ also.
This is very helpful in constructing gawk function libraries. If you have a large script with useful, general purposeawkfunctions, you can break it down into library files and put those filesin a special directory. You can then include those “libraries,” usingeither the full pathnames of the files, or by setting theAWKPATH
environment variable accordingly and then using ‘@include’ withjust the file part of the full pathname. Of course you can have morethan one directory to keep library files; the more complex the workingenvironment is, the more directories you may need to organize the filesto be included.
Given the ability to specify multiple -f options, the‘@include’ mechanism is not strictly necessary. However, the ‘@include’ keywordcan help you in constructing self-contained gawk programs,thus reducing the need for writing complex and tedious command lines. In particular, ‘@include’ is very useful for writing CGI scriptsto be run from web pages.
As mentioned in AWKPATH Variable, the current directory is alwayssearched first for source files, before searching inAWKPATH,and this also applies to files named with ‘@include’.
This section describes features and/or command-line options fromprevious releases of gawk that are either not available in thecurrent version or that are still supported but deprecated (meaning thatthey willnot be in the next release).
The process-related special files/dev/pid, /dev/ppid,/dev/pgrpid, and/dev/user were deprecated in gawk3.1, but still worked. As of version 4.0, they are no longerinterpreted specially bygawk. (Use PROCINFO
instead;seeAuto-set.)
Use the Source, Luke!
Obi-Wan
This section intentionally leftblank.
Aregular expression, or regexp, is a way of describing aset of strings. Because regular expressions are such a fundamental part ofawkprogramming, their format and use deserve a separate chapter.
A regular expression enclosed in slashes (‘/’)is anawk pattern that matches every input record whose textbelongs to that set. The simplest regular expression is a sequence of letters, numbers, orboth. Such a regexp matches any string that contains that sequence. Thus, the regexp ‘foo’ matches any string containing ‘foo’. Therefore, the pattern/foo/
matches any input record containingthe three characters ‘foo’anywhere in the record. Otherkinds of regexps let you specify more complicated classes of strings.
Initially, the examples in this chapter are simple. As we explain more about howregular expressions work, we present more complicated instances.
A regular expression can be used as a pattern by enclosing it inslashes. Then the regular expression is tested against theentire text of each record. (Normally, it only needsto match some part of the text in order to succeed.) For example, thefollowing prints the second field of each record that contains the string‘foo’ anywhere in it:
$ awk '/foo/ { print $2 }' BBS-list -| 555-1234 -| 555-6699 -| 555-6480 -| 555-2127
Regular expressions can also be used in matching expressions. Theseexpressions allow you to specify the string to match against; it neednot be the entire current input record. The two operators ‘~’and ‘!~’ perform regular expression comparisons. Expressionsusing these operators can be used as patterns, or inif
,while
, for
, and do
statements. (SeeStatements.) For example:
exp ~ /regexp/
is true if the expression exp (taken as a string)matchesregexp. The following example matches, or selects,all input records with the uppercase letter ‘J’ somewhere in thefirst field:
$ awk '$1 ~ /J/' inventory-shipped -| Jan 13 25 15 115 -| Jun 31 42 75 492 -| Jul 24 34 67 436 -| Jan 21 36 64 620
So does this:
awk '{ if ($1 ~ /J/) print }' inventory-shipped
This next example is true if the expression exp(taken as a character string)doesnot match regexp:
exp !~ /regexp/
The following example matches,or selects, all input records whose first field does not containthe uppercase letter ‘J’:
$ awk '$1 !~ /J/' inventory-shipped -| Feb 15 32 24 226 -| Mar 15 24 34 228 -| Apr 31 52 63 420 -| May 16 34 29 208 ...
When a regexp is enclosed in slashes, such as/foo/
, we call ita regexp constant, much like 5.27
is a numeric constant and"foo"
is a string constant.
Some characters cannot be included literally in string constants("foo"
) or regexp constants (/foo/
). Instead, they should be represented withescape sequences,which are character sequences beginning with a backslash (‘\’). One use of an escape sequence is to include a double-quote character ina string constant. Because a plain double quote ends the string, youmust use ‘\"’ to represent an actual double-quote character as apart of the string. For example:
$ awk 'BEGIN { print "He said \"hi!\" to her." }' -| He said "hi!" to her.
The backslash character itself is another character that cannot beincluded normally; you must write ‘\\’ to put one backslash in thestring or regexp. Thus, the string whose contents are the two characters‘"’ and ‘\’ must be written "\"\\"
.
Other escape sequences represent unprintable characterssuch as TAB or newline. While there is nothing to stop you from entering mostunprintable characters directly in a string constant or regexp constant,they may look ugly.
The following table listsall the escape sequences used in awk andwhat they represent. Unless noted otherwise, all these escapesequences apply to both string constants and regexp constants:
\\
\a
\b
\f
\n
\r
\t
\v
\
nnn
\x
hh
...
\/
\"
In gawk, a number of additional two-character sequences that beginwith a backslash have special meaning in regexps. SeeGNU Regexp Operators.
In a regexp, a backslash before any character that is not in the previous listand not listed inGNU Regexp Operators,means that the next character should be taken literally, even if it wouldnormally be a regexp operator. For example, /a\+b/
matches the threecharacters ‘a+b’.
For complete portability, do not use a backslash before any character notshown in the previous list.
To summarize:
If you place a backslash in a string constant before something that isnot one of the characters previously listed, POSIXawk purposelyleaves what happens as undefined. There are two choices:
"a\qc"
is the same as
"aqc"
. (Because this is such an easy bug both to introduce and to miss,
gawk warns you about it.) Consider ‘
FS = "[ \t]+\|[ \t]+"’ to use vertical barssurrounded by whitespace as the field separator. There should betwo backslashes in the string: ‘
FS = "[ \t]+\\|[ \t]+"’.)
"a\qc"
is the same as typing
"a\\qc"
.
Suppose you use an octal or hexadecimalescape to represent a regexp metacharacter. (SeeRegexp Operators.) Does awk treat the character as a literal character or as a regexpoperator?
Historically, such characters were taken literally. (d.c.) However, the POSIX standard indicates that they should be treatedas real metacharacters, which is whatgawk does. In compatibility mode (see Options),gawk treats the characters represented by octal and hexadecimalescape sequences literally when used in regexp constants. Thus,/a\52b/
is equivalent to/a\*b/
.
You can combine regular expressions with special characters,calledregular expression operators or metacharacters, toincrease the power and versatility of regular expressions.
The escape sequences describedearlierin Escape Sequences,are valid inside a regexp. They are introduced by a ‘\’ andare recognized and converted into corresponding real characters asthe very first step in processing regexps.
Here is a list of metacharacters. All characters that are not escapesequences and that are not listed in the table stand for themselves:
\
^
It is important to realize that ‘^’ does not match the beginning ofa line embedded in a string. The condition is not true in the following example:
if ("line1\nLINE 2" ~ /^L/) ...
$
if ("line1\nLINE 2" ~ /1$/) ...
.
(period)
In strict POSIX mode (seeOptions),‘.’ does not match thenulcharacter, which is a character with all bits equal to zero. Otherwise,nul is just another character. Other versions of awkmay not be able to match thenul character.
[...]
[^ ...]
|
The alternation applies to the largest possible regexps on either side.
(...)
*
The ‘*’ repeats the smallest possible preceding expression. (Use parentheses if you want to repeat a larger expression.) It findsas many repetitions as possible. For example,‘awk '/\(c[ad][ad]*r x\)/ { print }' sample’prints every record in sample containing a string of the form‘(car x)’, ‘(cdr x)’, ‘(cadr x)’, and so on. Notice the escaping of the parentheses by preceding themwith backslashes.
+
awk '/\(c[ad]+r x\)/ { print }' sample
?
{
n
}
{
n
,}
{
n
,
m
}
wh{3}y
wh{3,5}y
wh{2,}y
Interval expressions were not traditionally available inawk. They were added as part of the POSIX standard to makeawkand egrep consistent with each other.
Initially, because old programs may use ‘{’ and ‘}’ in regexpconstants,gawk did not match interval expressionsin regexps.
However, beginning with version 4.0,gawk does match interval expressions by default. This is because compatibility with POSIX has become moreimportant to mostgawk users than compatibility withold programs.
For programs that use ‘{’ and ‘}’ in regexp constants,it is good practice to always escape them with a backslash. Then theregexp constants are valid and work the way you want them to, usingany version of awk.16
In regular expressions, the ‘*’, ‘+’, and ‘?’ operators,as well as the braces ‘{’ and ‘}’,havethe highest precedence, followed by concatenation, and finally by ‘|’. As in arithmetic, parentheses can change how operators are grouped.
In POSIXawk and gawk, the ‘*’, ‘+’, and‘?’ operators stand for themselves when there is nothing in theregexp that precedes them. For example, /+/
matches a literalplus sign. However, many other versions ofawk treat such ausage as a syntax error.
If gawk is in compatibility mode (seeOptions), intervalexpressions are not available in regular expressions.
As mentioned earlier, a bracket expression matches any character amongstthose listed between the opening and closing square brackets.
Within a bracket expression, a range expression consists of twocharacters separated by a hyphen. It matches any single character thatsorts between the two characters, based upon the system's native characterset. For example, ‘[0-9]’ is equivalent to ‘[0123456789]’. (See Ranges and Locales, for an explanation of how the POSIXstandard and gawk have changed over time. This is mainlyof historical interest.)
To include one of the characters ‘\’, ‘]’, ‘-’, or ‘^’ in abracket expression, put a ‘\’ in front of it. For example:
[d\]]
matches either ‘d’ or ‘]’.
This treatment of ‘\’ in bracket expressionsis compatible with otherawkimplementations and is also mandated by POSIX. The regular expressions inawk are a supersetof the POSIX specification for Extended Regular Expressions (EREs). POSIX EREs are based on the regular expressions accepted by thetraditionalegrep utility.
Character classes are a feature introduced in the POSIX standard. A character class is a special notation for describinglists of characters that have a specific attribute, but theactual characters can vary from country to country and/orfrom character set to character set. For example, the notion of whatis an alphabetic character differs between the United States and France.
A character class is only valid in a regexp inside thebrackets of a bracket expression. Character classes consist of ‘[:’,a keyword denoting the class, and ‘:]’.table-char-classes lists the character classes defined by thePOSIX standard.
Class | Meaning |
---|---|
[:alnum:] |
Alphanumeric characters. |
[:alpha:] |
Alphabetic characters. |
[:blank:] |
Space and TAB characters. |
[:cntrl:] |
Control characters. |
[:digit:] |
Numeric characters. |
[:graph:] |
Characters that are both printable and visible. (A space is printable but not visible, whereas an ‘a’ is both.) |
[:lower:] |
Lowercase alphabetic characters. |
[:print:] |
Printable characters (characters that are not control characters). |
[:punct:] |
Punctuation characters (characters that are not letters, digits,control characters, or space characters). |
[:space:] |
Space characters (such as space, TAB, and formfeed, to name a few). |
[:upper:] |
Uppercase alphabetic characters. |
[:xdigit:] |
Characters that are hexadecimal digits. |
Table 3.1: POSIX Character Classes
For example, before the POSIX standard, you had to write /[A-Za-z0-9]/
to match alphanumeric characters. If yourcharacter set had other alphabetic characters in it, this would notmatch them. With the POSIX character classes, you can write/[[:alnum:]]/
to match the alphabeticand numeric characters in your character set.
Two additional special sequences can appear in bracket expressions. These apply to non-ASCII character sets, which can have single symbols(called collating elements) that are represented with more than onecharacter. They can also have several characters that are equivalent forcollating, or sorting, purposes. (For example, in French, a plain “e”and a grave-accented “è” are equivalent.) These sequences are:
[[.ch.]]
is a regexp that matches this collating element, whereas
[ch]
is a regexp that matches either ‘
c’ or ‘
h’.
[[=e=]]
is a regexpthat matches any of ‘
e’, ‘
é’, or ‘
è’.
These features are very valuable in non-English-speaking locales.
CAUTION: The library functions that gawk uses for regularexpression matching currently recognize only POSIX character classes;they do not recognize collating symbols or equivalence classes.
GNU software that deals with regular expressions provides a number ofadditional regexp operators. These operators are described in thissection and are specific togawk;they are not available in other awk implementations. Most of the additional operators deal with word matching. For our purposes, aword is a sequence of one or more letters, digits,or underscores (‘_’):
\s
[[:space:]]
.
\S
[^[:space:]]
.
\w
[[:alnum:]_]
.
\W
[^[:alnum:]_]
.
\<
/\<away/
matches ‘
away’ but not‘
stowaway’. <!-- @cindex operators, @code{\>} (@command{gawk}) -->
\>
/stow\>/
matches ‘
stow’ but not ‘
stowaway’.
\y
\B
/\Brat\B/
matches ‘
crate’ but it does not match ‘
dirty rat’. ‘
\B’ is essentially the opposite of ‘
\y’.
There are two other operators that work on buffers. In Emacs, abuffer is, naturally, an Emacs buffer. For other programs,gawk's regexp library routines consider the entirestring to match as the buffer. The operators are:
\`
\'
Because ‘^’ and ‘$’ always work in terms of the beginningand end of strings, these operators don't add any new capabilitiesforawk. They are provided for compatibility with otherGNU software.
In other GNU software, the word-boundary operator is ‘\b’. However,that conflicts with theawk language's definition of ‘\b’as backspace, sogawk uses a different letter. An alternative method would have been to require two backslashes in theGNU operators, but this was deemed too confusing. The currentmethod of using ‘\y’ for the GNU ‘\b’ appears to be thelesser of two evils.
The various command-line options(seeOptions)control how gawk interprets characters in regexps:
--posix
--traditional
[[:alnum:]]
, etc.) are supported,as Brian Kernighan's
awk does support them. Characters described by octal and hexadecimal escape sequences aretreated literally, even if they represent regexp metacharacters.
--re-interval
Case is normally significant in regular expressions, both when matchingordinary characters (i.e., not metacharacters) and inside bracketexpressions. Thus, a ‘w’ in a regular expression matches only a lowercase‘w’ and not an uppercase ‘W’.
The simplest way to do a case-independent match is to use a bracketexpression—for example, ‘[Ww]’. However, this can be cumbersome ifyou need to use it often, and it can make the regular expressions harderto read. There are two alternatives that you might prefer.
One way to perform a case-insensitive match at a particular point in theprogram is to convert the data to a single case, using thetolower()
ortoupper()
built-in string functions (which wehaven't discussed yet;seeString Functions). For example:
tolower($1) ~ /foo/ { ... }
converts the first field to lowercase before matching against it. This works in any POSIX-compliantawk.
Another method, specific to gawk, is to set the variableIGNORECASE
to a nonzero value (seeBuilt-in Variables). When IGNORECASE
is not zero,all regexp and stringoperations ignore case. Changing the value ofIGNORECASE
dynamically controls the case-sensitivity of theprogram as it runs. Case is significant by default becauseIGNORECASE
(like most variables) is initialized to zero:
x = "aB" if (x ~ /ab/) ... # this test will fail IGNORECASE = 1 if (x ~ /ab/) ... # now it will succeed
In general, you cannot use IGNORECASE
to make certain rulescase-insensitive and other rules case-sensitive, because there is nostraightforward wayto setIGNORECASE
just for the pattern ofa particular rule.17To do this, use either bracket expressions ortolower()
. However, onething you can do with IGNORECASE
only is dynamically turncase-sensitivity on or off for all the rules at once.
IGNORECASE
can be set on the command line or in a BEGIN
rule(seeOther Arguments; alsosee Using BEGIN/END). Setting IGNORECASE
from the command line is a way to makea program case-insensitive without having to edit it.
Both regexp and string comparisonoperations are affected by IGNORECASE
.
In multibyte locales,the equivalences between upper-and lowercase characters are tested based on the wide-character values ofthe locale's character set. Otherwise, the characters are tested basedon the ISO-8859-1 (ISO Latin-1)character set. This character set is a superset of the traditional 128ASCII characters, which also provides a number of characters suitablefor use with European languages.18
The value of IGNORECASE
has no effect if gawk is incompatibility mode (seeOptions). Case is always significant in compatibility mode.
Consider the following:
echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }'
This example uses the sub()
function (which we haven't discussed yet;seeString Functions)to make a change to the input record. Here, the regexp/a+/
indicates “one or more ‘a’ characters,” and the replacementtext is ‘<A>’.
The input contains four ‘a’ characters.awk (and POSIX) regular expressions always matchthe leftmost,longest sequence of input characters that canmatch. Thus, all four ‘a’ characters arereplaced with ‘<A>’ in this example:
$ echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }' -| <A>bcd
For simple match/no-match tests, this is not so important. But when doingtext matching and substitutions with thematch()
, sub()
, gsub()
,and gensub()
functions, it is very important. Understanding this principle is also important for regexp-based recordand field splitting (seeRecords,and also see Field Separators).
The righthand side of a ‘~’ or ‘!~’ operator need not be aregexp constant (i.e., a string of characters between slashes). It maybe any expression. The expression is evaluated and converted to a stringif necessary; the contents of the string are then used as theregexp. A regexp computed in this way is called adynamicregexp:
BEGIN { digits_regexp = "[[:digit:]]+" } $0 ~ digits_regexp { print }
This sets digits_regexp
to a regexp that describes one or more digits,and tests whether the input record matches this regexp.
NOTE: When using the ‘
~’ and ‘
!~’operators, there is a difference between a regexp constantenclosed in slashes and a string constant enclosed in double quotes. If you are going to use a string constant, you have to understand thatthe string is, in essence, scanned
twice: the first time when
awk reads your program, and the second time when it goes tomatch the string on the lefthand side of the operator with the patternon the right. This is true of any string-valued expression (such as
digits_regexp
, shown previously), not just string constants.
What difference does it make if the string isscanned twice? The answer has to do with escape sequences, and particularlywith backslashes. To get a backslash into a regular expression inside astring, you have to type two backslashes.
For example, /\*/
is a regexp constant for a literal ‘*’. Only one backslash is needed. To do the same thing with a string,you have to type"\\*"
. The first backslash escapes thesecond one so that the string actually contains thetwo characters ‘\’ and ‘*’.
Given that you can use both regexp and string constants to describeregular expressions, which should you use? The answer is “regexpconstants,” for several reasons:
\n
in Bracket Expressions of Dynamic RegexpsSome commercial versions ofawk do not allow the newlinecharacter to be used inside a bracket expression for a dynamic regexp:
$ awk '$0 ~ "[ \t\n]"' error--> awk: newline in character class [ error--> ]... error--> source line number 1 error--> context is error--> >>> <<<
But a newline in a regexp constant works with no problem:
$ awk '$0 ~ /[ \t\n]/' here is a sample line -| here is a sample line Ctrl-d
gawk does not have this problem, and it isn't likely tooccur often in practice, but it's worth noting for future reference.
In the typicalawk program,awk reads all input either from thestandard input (by default, this is the keyboard, but often it is a pipe from anothercommand) or from files whose names you specify on the awkcommand line. If you specify input files,awk reads themin order, processing all the data from one before going on to the next. The name of the current input file can be found in the built-in variableFILENAME
(seeBuilt-in Variables).
The input is read in units calledrecords, and is processed by therules of your program one record at a time. By default, each record is one line. Eachrecord is automatically split into chunks calledfields. This makes it more convenient for programs to work on the parts of a record.
On rare occasions, you may need to use thegetline
command. The getline
command is valuable, both because itcan do explicit input from any number of files, and because the filesused with it do not have to be named on theawk command line(see Getline).
Theawk utility divides the input for your awkprogram into records and fields. awk keeps track of the number of records that havebeen readso farfrom the current input file. This value is stored in abuilt-in variable calledFNR
. It is reset to zero when a newfile is started. Another built-in variable,NR
, records the totalnumber of input records read so far from all data files. It starts at zero,but is never automatically reset to zero.
Records are separated by a character called therecord separator. By default, the record separator is the newline character. This is why records are, by default, single lines. A different character can be used for the record separator byassigning the character to the built-in variableRS
.
Like any other variable,the value ofRS
can be changed in the awk programwith the assignment operator, ‘=’(seeAssignment Ops). The new record-separator character should be enclosed in quotation marks,which indicate a string constant. Often the right time to do this isat the beginning of execution, before any input is processed,so that the very first record is read with the proper separator. To do this, use the specialBEGIN
pattern(see BEGIN/END). For example:
awk 'BEGIN { RS = "/" } { print $0 }' BBS-list
changes the value of RS
to "/"
, before reading any input. This is a string whose first character is a slash; as a result, recordsare separated by slashes. Then the input file is read, and the secondrule in theawk program (the action with no pattern) prints eachrecord. Because eachprint
statement adds a newline at the end ofits output, this awk program copies the inputwith each slash changed to a newline. Here are the results of runningthe program onBBS-list:
$ awk 'BEGIN { RS = "/" } > { print $0 }' BBS-list -| aardvark 555-5553 1200 -| 300 B -| alpo-net 555-3412 2400 -| 1200 -| 300 A -| barfly 555-7685 1200 -| 300 A -| bites 555-1675 2400 -| 1200 -| 300 A -| camelot 555-0542 300 C -| core 555-2912 1200 -| 300 C -| fooey 555-1234 2400 -| 1200 -| 300 B -| foot 555-6699 1200 -| 300 B -| macfoo 555-6480 1200 -| 300 A -| sdace 555-3430 2400 -| 1200 -| 300 A -| sabafoo 555-2127 1200 -| 300 C -|
Note that the entry for the ‘camelot’ BBS is not split. In the original data file(seeSample Data Files),the line looks like this:
camelot 555-0542 300 C
It has one baud rate only, so there are no slashes in the record,unlike the others which have two or more baud rates. In fact, this record is treated as part of the recordfor the ‘core’ BBS; the newline separating them in the outputis the original newline in the data file, not the one added byawk when it printed the record!
Another way to change the record separator is on the command line,using the variable-assignment feature(seeOther Arguments):
awk '{ print $0 }' RS="/" BBS-list
This sets RS
to ‘/’ before processingBBS-list.
Using an unusual character such as ‘/’ for the record separatorproduces correct behavior in the vast majority of cases. However,the following (extreme) pipeline prints a surprising ‘1’:
$ echo | awk 'BEGIN { RS = "a" } ; { print NF }' -| 1
There is one field, consisting of a newline. The value of the built-invariableNF
is the number of fields in the current record.
Reaching the end of an input file terminates the current input record,even if the last character in the file is not the character inRS
. (d.c.)
The empty string""
(a string without any characters)has a special meaningas the value ofRS
. It means that records are separatedby one or more blank lines and nothing else. SeeMultiple Line, for more details.
If you change the value of RS
in the middle of an awk run,the new value is used to delimit subsequent records, but the recordcurrently being processed, as well as records already processed, are notaffected.
After the end of the record has been determined, gawksets the variableRT
to the text in the input that matchedRS
.
When usinggawk,the value of RS
is not limited to a one-characterstring. It can be any regular expression(seeRegexp). (c.e.) In general, each recordends at the next string that matches the regular expression; the nextrecord starts at the end of the matching string. This general rule isactually at work in the usual case, whereRS
contains just anewline: a record ends at the beginning of the next matching string (thenext newline in the input), and the following record starts just afterthe end of this string (at the first character of the following line). The newline, because it matches RS
, is not part of either record.
When RS
is a single character, RT
contains the same single character. However, whenRS
is aregular expression, RT
containsthe actual input text that matched the regular expression.
If the input file ended without any text that matches RS
,gawk setsRT
to the null string.
The following example illustrates both of these features. It sets RS
equal to a regular expression thatmatches either a newline or a series of one or more uppercase letterswith optional leading and/or trailing whitespace:
$ echo record 1 AAAA record 2 BBBB record 3 | > gawk 'BEGIN { RS = "\n|( *[[:upper:]]+ *)" } > { print "Record =", $0, "and RT =", RT }' -| Record = record 1 and RT = AAAA -| Record = record 2 and RT = BBBB -| Record = record 3 and RT = -|
The final line of output has an extra blank line. This is because thevalue ofRT
is a newline, and the print
statementsupplies its own terminating newline. SeeSimple Sed, for a more useful exampleof RS
as a regexp andRT
.
If you set RS
to a regular expression that allows optionaltrailing text, such as ‘RS = "abc(XYZ)?"’ it is possible, dueto implementation constraints, thatgawk may match the leadingpart of the regular expression, but not the trailing part, particularlyif the input text that could match the trailing part is fairly long.gawk attempts to avoid this problem, but currently, there'sno guarantee that this will never happen.
NOTE: Remember that in
awk, the ‘
^’ and ‘
$’ anchormetacharacters match the beginning and end of a
string, and notthe beginning and end of a
line. As a result, something like‘
RS = "^[[:upper:]]"’ can only match at the beginning of a file. This is because
gawk views the input file as one long stringthat happens to contain newline characters in it. It is thus best to avoid anchor characters in the value of
RS
.
The use ofRS
as a regular expression and the RT
variable are gawk extensions; they are not available incompatibility mode(seeOptions). In compatibility mode, only the first character of the value ofRS
is used to determine the end of the record.
RS = "\0"
Is Not PortableThere are times when you might want to treat an entire data file as asingle record. The only way to make this happen is to give RS
a value that you know doesn't occur in the input file. This is hardto do in a general way, such that a program always works for arbitraryinput files.
You might think that for text files, the nul character, whichconsists of a character with all bits equal to zero, is a goodvalue to use forRS
in this case:
BEGIN { RS = "\0" } # whole file becomes one record?
gawk in fact accepts this, and uses thenulcharacter for the record separator. However, this usage isnot portableto other awk implementations.
All other awk implementations19 store strings internally as C-style strings. C strings use thenul character as the string terminator. In effect, this means that‘RS = "\0"’ is the same as ‘RS = ""’. (d.c.)
The best way to treat a whole file as a single record is tosimply read the file in, one record at a time, concatenating eachrecord onto the end of the previous ones.
Whenawk reads an input record, the record isautomaticallyparsed or separated by the awk utility into chunkscalledfields. By default, fields are separated by whitespace,like words in a line. Whitespace inawk means any string of one or more spaces,TABs, or newlines;20 other characters, such asformfeed, vertical tab, etc., that areconsidered whitespace by other languages, are not consideredwhitespace by awk.
The purpose of fields is to make it more convenient for you to refer tothese pieces of the record. You don't have to use them—you canoperate on the whole record if you want—but fields are what makesimpleawk programs so powerful.
A dollar-sign (‘$’) is usedto refer to a field in anawk program,followed by the number of the field you want. Thus,$1
refers to the first field, $2
to the second, and so on. (Unlike the Unix shells, the field numbers are not limited to single digits.$127
is the one hundred twenty-seventh field in the record.) For example, suppose the following is a line of input:
This seems like a pretty nice example.
Here the first field, or $1
, is ‘This’, the second field, or$2
, is ‘seems’, and so on. Note that the last field,$7
, is ‘example.’. Because there is no space between the‘e’ and the ‘.’, the period is considered part of the seventhfield.
NF
is a built-in variable whose value is the number of fieldsin the current record.awk automatically updates the valueof NF
each time it reads a record. No matter how many fieldsthere are, the last field in a record can be represented by$NF
. So, $NF
is the same as $7
, which is ‘example.’. If you try to reference a field beyond the lastone (such as$8
when the record has only seven fields), you getthe empty string. (If used in a numeric operation, you get zero.)
The use of $0
, which looks like a reference to the “zero-th” field, isa special case: it represents the whole input recordwhen you are not interested in specific fields. Here are some more examples:
$ awk '$1 ~ /foo/ { print $0 }' BBS-list -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sabafoo 555-2127 1200/300 C
This example prints each record in the file BBS-list whose firstfield contains the string ‘foo’. The operator ‘~’ is called amatching operator(see Regexp Usage);it tests whether a string (here, the field$1
) matches a given regularexpression.
By contrast, the following examplelooks for ‘foo’ inthe entire record and prints the firstfield and the last field for each matching input record:
$ awk '/foo/ { print $1, $NF }' BBS-list -| fooey B -| foot B -| macfoo A -| sabafoo C
The number of a field does not need to be a constant. Any expression intheawk language can be used after a ‘$’ to refer to afield. The value of the expression specifies the field number. If thevalue is a string, rather than a number, it is converted to a number. Consider this example:
awk '{ print $NR }'
Recall that NR
is the number of records read so far: one in thefirst record, two in the second, etc. So this example prints the firstfield of the first record, the second field of the second record, and soon. For the twentieth record, field number 20 is printed; most likely,the record has fewer than 20 fields, so this prints a blank line. Here is another example of using expressions as field numbers:
awk '{ print $(2*2) }' BBS-list
awk evaluates the expression ‘(2*2)’ and usesits value as the number of the field to print. The ‘*’ signrepresents multiplication, so the expression ‘2*2’ evaluates to four. The parentheses are used so that the multiplication is done before the‘$’ operation; they are necessary whenever there is a binaryoperator in the field-number expression. This example, then, prints thehours of operation (the fourth field) for every line of the fileBBS-list. (All of theawk operators are listed, inorder of decreasing precedence, inPrecedence.)
If the field number you compute is zero, you get the entire record. Thus, ‘$(2-2)’ has the same value as$0
. Negative fieldnumbers are not allowed; trying to reference one usually terminatesthe program. (The POSIX standard does not definewhat happens when you reference a negative field number.gawknotices this and terminates your program. Otherawkimplementations may behave differently.)
As mentioned in Fields,awk stores the current record's number of fields in the built-invariableNF
(also see Built-in Variables). The expression$NF
is not a special feature—it is the direct consequence ofevaluatingNF
and using its value as a field number.
The contents of a field, as seen byawk, can be changed within anawk program; this changes whatawk perceives as thecurrent input record. (The actual input is untouched;awk nevermodifies the input file.) Consider the following example and its output:
$ awk '{ nboxes = $3 ; $3 = $3 - 10 > print nboxes, $3 }' inventory-shipped -| 25 15 -| 32 22 -| 24 14 ...
The program first saves the original value of field three in the variablenboxes
. The ‘-’ sign represents subtraction, so this program reassignsfield three,$3
, as the original value of field three minus ten:‘$3 - 10’. (SeeArithmetic Ops.) Then it prints the original and new values for field three. (Someone in the warehouse made a consistent mistake while inventoryingthe red boxes.)
For this to work, the text in field $3
must make senseas a number; the string of characters must be converted to a numberfor the computer to do arithmetic on it. The number resultingfrom the subtraction is converted back to a string of characters thatthen becomes field three. See Conversion.
When the value of a field is changed (as perceived by awk), thetext of the input record is recalculated to contain the new field wherethe old one was. In other words,$0
changes to reflect the alteredfield. Thus, this programprints a copy of the input file, with 10 subtracted from the secondfield of each line:
$ awk '{ $2 = $2 - 10; print $0 }' inventory-shipped -| Jan 3 25 15 115 -| Feb 5 32 24 226 -| Mar 5 24 34 228 ...
It is also possible to also assign contents to fields that are outof range. For example:
$ awk '{ $6 = ($5 + $4 + $3 + $2) > print $6 }' inventory-shipped -| 168 -| 297 -| 301 ...
We've just created$6
, whose value is the sum of fields$2
, $3
,$4
, and $5
. The ‘+’ signrepresents addition. For the fileinventory-shipped, $6
represents the total number of parcels shipped for a particular month.
Creating a new field changes awk's internal copy of the currentinput record, which is the value of$0
. Thus, if you do ‘print $0’after adding a field, the record printed includes the new field, withthe appropriate number of field separators between it and the previouslyexisting fields.
This recomputation affects and is affected byNF
(the number of fields; see Fields). For example, the value ofNF
is set to the number of the highestfield you create. The exact format of$0
is also affected by a feature that has not been discussed yet:theoutput field separator, OFS
,used to separate the fields (seeOutput Separators).
Note, however, that merely referencing an out-of-range fielddoes not change the value of either $0
or NF
. Referencing an out-of-range field only produces an empty string. Forexample:
if ($(NF+1) != "") print "can't happen" else print "everything is normal"
should print ‘everything is normal’, becauseNF+1
is certainto be out of range. (See If Statement,for more information aboutawk's if-else
statements. SeeTyping and Comparison,for more information about the ‘!=’ operator.)
It is important to note that making an assignment to an existing fieldchanges thevalue of$0
but does not change the value of NF
,even when you assign the empty string to a field. For example:
$ echo a b c d | awk '{ OFS = ":"; $2 = "" > print $0; print NF }' -| a::c:d -| 4
The field is still there; it just has an empty value, denoted bythe two colons between ‘a’ and ‘c’. This example shows what happens if you create a new field:
$ echo a b c d | awk '{ OFS = ":"; $2 = ""; $6 = "new" > print $0; print NF }' -| a::c:d::new -| 6
The intervening field, $5
, is created with an empty value(indicated by the second pair of adjacent colons),andNF
is updated with the value six.
DecrementingNF
throws away the values of the fieldsafter the new value of NF
and recomputes $0
. (d.c.) Here is an example:
$ echo a b c d e f | awk '{ print "NF =", NF; > NF = 3; print $0 }' -| NF = 6 -| a b c
CAUTION: Some versions of awk don'trebuild$0
whenNF
is decremented. Caveat emptor.
Finally, there are times when it is convenient to forceawk to rebuild the entire record, using the currentvalue of the fields andOFS
. To do this, use theseemingly innocuous assignment:
$1 = $1 # force record to be reconstituted print $0 # or whatever else with $0
This forces awk rebuild the record. It does helpto add a comment, as we've shown here.
There is a flip side to the relationship between $0
andthe fields. Any assignment to$0
causes the record to bereparsed into fields using the current value ofFS
. This also applies to any built-in function that updates $0
,such assub()
and gsub()
(see String Functions).
$0
It is important to remember that $0
is the fullrecord, exactly as it was read from the input. This includesany leading or trailing whitespace, and the exact whitespace (or othercharacters) that separate the fields.
It is a not-uncommon error to try to change the field separatorsin a record simply by settingFS
and OFS
, and thenexpecting a plain ‘print’ or ‘print $0’ to print themodified record.
But this does not work, since nothing was done to change the recorditself. Instead, you must force the record to be rebuilt, typicallywith a statement such as ‘$1 = $1’, as described earlier.
Thefield separator, which is either a single character or a regularexpression, controls the wayawk splits an input record into fields.awk scans the input record for character sequences thatmatch the separator; the fields themselves are the text between the matches.
In the examples that follow, we use the bullet symbol (•) torepresent spaces in the output. If the field separator is ‘oo’, then the following line:
moo goo gai pan
is split into three fields: ‘m’, ‘•g’, and‘•gai•pan’. Note the leading spaces in the values of the second and third fields.
The field separator is represented by the built-in variableFS
. Shell programmers take note: awk doesnot use thename IFS
that is used by the POSIX-compliant shells (such asthe Unix Bourne shell,sh, or Bash).
The value ofFS
can be changed in the awk program with theassignment operator, ‘=’ (seeAssignment Ops). Often the right time to do this is at the beginning of executionbefore any input has been processed, so that the very first recordis read with the proper separator. To do this, use the specialBEGIN
pattern(see BEGIN/END). For example, here we set the value ofFS
to the string","
:
awk 'BEGIN { FS = "," } ; { print $2 }'
Given the input line:
John Q. Smith, 29 Oak St., Walamazoo, MI 42139
this awk program extracts and prints the string‘•29•Oak•St.’.
Sometimes the input data contains separator characters that don'tseparate fields the way you thought they would. For instance, theperson's name in the example we just used might have a title orsuffix attached, such as:
John Q. Smith, LXIX, 29 Oak St., Walamazoo, MI 42139
The same program would extract ‘•LXIX’, instead of‘•29•Oak•St.’. If you were expecting the program to print theaddress, you would be surprised. The moral is to choose your data layout andseparator characters carefully to prevent such problems. (If the data is not in a form that is easy to process, perhaps youcan massage it first with a separateawk program.)
Fields are normally separated by whitespace sequences(spaces, TABs, and newlines), not by single spaces. Two spaces in a row do notdelimit an empty field. The default value of the field separator FS
is a string containing a single space," "
. If awkinterpreted this value in the usual way, each space character would separatefields, so two spaces in a row would make an empty field between them. The reason this does not happen is that a single space as the value ofFS
is a special case—it is taken to specify the default mannerof delimiting fields.
If FS
is any other single character, such as ","
, theneach occurrence of that character separates two fields. Two consecutiveoccurrences delimit an empty field. If the character occurs at thebeginning or the end of the line, that too delimits an empty field. Thespace character is the only single character that does not follow theserules.
The previous subsectiondiscussed the use of single characters or simple strings as thevalue ofFS
. More generally, the value of FS
may be a string containing anyregular expression. In this case, each match in the record for the regularexpression separates fields. For example, the assignment:
FS = ", \t"
makes every area of an input line that consists of a comma followed by aspace and a TAB into a field separator.
For a less trivial example of a regular expression, try usingsingle spaces to separate fields the way single commas are used.FS
can be set to "[ ]"
(left bracket, space, rightbracket). This regular expression matches a single space and nothing else(seeRegexp).
There is an important difference between the two cases of ‘FS = " "’(a single space) and ‘FS = "[ \t\n]+"’(a regular expression matching one or more spaces, TABs, or newlines). For both values of FS
, fields are separated by runs(multiple adjacent occurrences) of spaces, TABs,and/or newlines. However, when the value ofFS
is " "
,awk first strips leading and trailing whitespace fromthe record and then decides where the fields are. For example, the following pipeline prints ‘b’:
$ echo ' a b c d ' | awk '{ print $2 }' -| b
However, this pipeline prints ‘a’ (note the extra spaces aroundeach letter):
$ echo ' a b c d ' | awk 'BEGIN { FS = "[ \t\n]+" } > { print $2 }' -| a
In this case, the first field isnull or empty.
The stripping of leading and trailing whitespace also comes intoplay whenever $0
is recomputed. For instance, study this pipeline:
$ echo ' a b c d' | awk '{ print; $2 = $2; print }' -| a b c d -| a b c d
The first print
statement prints the record as it was read,with leading whitespace intact. The assignment to$2
rebuilds$0
by concatenating $1
through $NF
together,separated by the value of OFS
. Because the leading whitespacewas ignored when finding$1
, it is not part of the new $0
. Finally, the last print
statement prints the new $0
.
There is an additional subtlety to be aware of when using regular expressionsfor field splitting. It is not well-specified in the POSIX standard, or anywhere else, what ‘^’means when splitting fields. Does the ‘^’ match only at the beginning ofthe entire record? Or is each field separator a new string? It turns out thatdifferentawk versions answer this question differently, and youshould not rely on any specific behavior in your programs. (d.c.)
As a point of information, Brian Kernighan's awk allows ‘^’to match only at the beginning of the record.gawkalso works this way. For example:
$ echo 'xxAA xxBxx C' | > gawk -F '(^x+)|( +)' '{ for (i = 1; i <= NF; i++) > printf "-->%s<--\n", $i }' -| --><-- -| -->AA<-- -| -->xxBxx<-- -| -->C<--
There are times when you may want to examine each characterof a record separately. This can be done ingawk bysimply assigning the null string (""
) toFS
. (c.e.) In this case,each individual character in the record becomes a separate field. For example:
$ echo a b | gawk 'BEGIN { FS = "" } > { > for (i = 1; i <= NF; i = i + 1) > print "Field", i, "is", $i > }' -| Field 1 is a -| Field 2 is -| Field 3 is b
Traditionally, the behavior ofFS
equal to ""
was not defined. In this case, most versions of Unixawk simply treat the entire recordas only having one field. (d.c.) In compatibility mode(seeOptions),if FS
is the null string, then gawk alsobehaves this way.
FS
from the Command LineFS
can be set on the command line. Use the -F option todo so. For example:
awk -F, 'program' input-files
sets FS
to the ‘,’ character. Notice that the option usesan uppercase ‘F’ instead of a lowercase ‘f’. The latteroption (-f) specifies a filecontaining an awk program. Case is significant in command-lineoptions:the-F and -f options have nothing to do with each other. You can use both options at the same time to set theFS
variableand get an awk program from a file.
The value used for the argument to -F is processed in exactly thesame way as assignments to the built-in variableFS
. Any special characters in the field separator must be escapedappropriately. For example, to use a ‘\’ as the field separatoron the command line, you would have to type:
# same as FS = "\\" awk -F\\\\ '...' files ...
Because ‘\’ is used for quoting in the shell, awk sees‘-F\\’. Thenawk processes the ‘\\’ for escapecharacters (seeEscape Sequences), finally yieldinga single ‘\’ to use for the field separator.
As a special case, in compatibility mode(see Options),if the argument to-F is ‘t’, thenFS
is set tothe TAB character. If you type ‘-F\t’ at theshell, without any quotes, the ‘\’ gets deleted, soawkfigures that you really want your fields to be separated with TABs andnot ‘t’s. Use ‘-v FS="t"’ or ‘-F"[t]"’ on the command lineif you really do want to separate your fields with ‘t’s.
As an example, let's use an awk program file calledbaud.awkthat contains the pattern /300/
and the action ‘print $1’:
/300/ { print $1 }
Let's also set FS
to be the ‘-’ character and run theprogram on the fileBBS-list. The following command prints alist of the names of the bulletin boards that operate at 300 baud andthe first three digits of their phone numbers:
$ awk -F- -f baud.awk BBS-list -| aardvark 555 -| alpo -| barfly 555 -| bites 555 -| camelot 555 -| core 555 -| fooey 555 -| foot 555 -| macfoo 555 -| sdace 555 -| sabafoo 555
Note the second line of output. The second linein the original file looked like this:
alpo-net 555-3412 2400/1200/300 A
The ‘-’ as part of the system's name was used as the fieldseparator, instead of the ‘-’ in the phone number that wasoriginally intended. This demonstrates why you have to be careful inchoosing your field and record separators.
Perhaps the most common use of a single character as the fieldseparator occurs when processing the Unix system password file. On many Unix systems, each user has a separate entry in the system passwordfile, one line per user. The information in these lines is separatedby colons. The first field is the user's login name and the second isthe user's (encrypted or shadow) password. A password file entry might looklike this:
arnold:xyzzy:2076:10:Arnold Robbins:/home/arnold:/bin/bash
The following program searches the system password file and printsthe entries for users who have no password:
awk -F: '$2 == ""' /etc/passwd
It is important to remember that when you assign a string constantas the value ofFS
, it undergoes normal awk stringprocessing. For example, with Unixawk and gawk,the assignment ‘FS = "\.."’ assigns the character string".."
to FS
(the backslash is stripped). This creates a regexp meaning“fields are separated by occurrences of any two characters.”If instead you want fields to be separated by a literal period followedby any single character, use ‘FS = "\\.."’.
The following table summarizes how fields are split, based on the valueof FS
(‘==’ means “is equal to”):
FS == " "
FS ==
any other single character
FS ==
regexp
FS == ""
FS
Does Not Affect the FieldsAccording to the POSIX standard,awk is supposed to behaveas if each record is split into fields at the time it is read. In particular, this means that if you change the value ofFS
after a record is read, the value of the fields (i.e., how they were split)should reflect the old value ofFS
, not the new one.
However, many older implementations ofawk do not work this way. Instead,they defer splitting the fields until a field is actuallyreferenced. The fields are splitusing thecurrent value of FS
! (d.c.) This behavior can be difficultto diagnose. The following example illustrates the differencebetween the two methods. (Thesed21command prints just the first line of/etc/passwd.)
sed 1q /etc/passwd | awk '{ FS = ":" ; print $1 }'
which usually prints:
root
on an incorrect implementation of awk, whilegawkprints something like:
root:nSijPlPhZZwgE:0:0:Root:/:
FS
and IGNORECASE
The IGNORECASE
variable(see User-modified)affects field splittingonly when the value of FS
is a regexp. It has no effect whenFS
is a single character, even ifthat character is a letter. Thus, in the following code:
FS = "c" IGNORECASE = 1 $0 = "aCa" print $1
The output is ‘aCa’. If you really want to split fields on analphabetic character while ignoring case, use a regexp that willdo it for you. E.g., ‘FS = "[c]"’. In this case, IGNORECASE
will take effect.
NOTE: This section discusses an advancedfeature of gawk. If you are a novice awk user,you might want to skip it on the first reading.
gawk provides a facility for dealing withfixed-width fields with no distinctive field separator. For example,data of this nature arises in the input for old Fortran programs wherenumbers are run together, or in the output of programs that did notanticipate the use of their output as input for other programs.
An example of the latter is a table where all the columns are lined up bythe use of a variable number of spaces andempty fields are justspaces. Clearly, awk's normal field splitting based onFS
does not work well in this case. Although a portable awk programcan use a series ofsubstr()
calls on $0
(see String Functions),this is awkward and inefficient for a large number of fields.
The splitting of an input record into fixed-width fields is specified byassigning a string containing space-separated numbers to the built-invariableFIELDWIDTHS
. Each number specifies the width of the field,including columns between fields. If you want to ignore the columnsbetween fields, you can specify the width as a separate field that issubsequently ignored. It is a fatal error to supply a field width that is not a positive number. The following data is the output of the Unixw utility. It is usefulto illustrate the use ofFIELDWIDTHS
:
10:06pm up 21 days, 14:04, 23 users User tty login idle JCPU PCPU what hzuo ttyV0 8:58pm 9 5 vi p24.tex hzang ttyV3 6:37pm 50 -csh eklye ttyV5 9:53pm 7 1 em thes.tex dportein ttyV6 8:17pm 1:47 -csh gierd ttyD3 10:00pm 1 elm dave ttyD4 9:47pm 4 4 w brent ttyp0 26Jun91 4:46 26:46 4:41 bash dave ttyq4 26Jun9115days 46 46 wnewmail
The following program takes the above input, converts the idle time tonumber of seconds, and prints out the first two fields and the calculatedidle time:
NOTE: This program uses a number of awk features thathaven't been introduced yet.
BEGIN { FIELDWIDTHS = "9 6 10 6 7 7 35" } NR > 2 { idle = $4 sub(/^ */, "", idle) # strip leading spaces if (idle == "") idle = 0 if (idle ~ /:/) { split(idle, t, ":") idle = t[1] * 60 + t[2] } if (idle ~ /days/) idle *= 24 * 60 * 60 print $1, $2, idle }
Running the program on the data produces the following results:
hzuo ttyV0 0 hzang ttyV3 50 eklye ttyV5 0 dportein ttyV6 107 gierd ttyD3 1 dave ttyD4 0 brent ttyp0 286 dave ttyq4 1296000
Another (possibly more practical) example of fixed-width input datais the input from a deck of balloting cards. In some parts ofthe United States, voters mark their choices by punching holes in computercards. These cards are then processed to count the votes for any particularcandidate or on any particular issue. Because a voter may choose not tovote on some issue, any column on the card may be empty. Anawkprogram for processing such data could use theFIELDWIDTHS
featureto simplify reading the data. (Of course, gettinggawk to run ona system with card readers is another story!)
Assigning a value toFS
causes gawk to useFS
for field splitting again. Use ‘FS = FS’ to make this happen,without having to know the current value ofFS
. In order to tell which kind of field splitting is in effect,use PROCINFO["FS"]
(see Auto-set). The value is "FS"
if regular field splitting is being used,or it is "FIELDWIDTHS"
if fixed-width field splitting is being used:
if (PROCINFO["FS"] == "FS") regular field splitting ... else if (PROCINFO["FS"] == "FIELDWIDTHS") fixed-width field splitting ... else content-based field splitting ... (see next section)
This information is useful when writing a functionthat needs to temporarily changeFS
or FIELDWIDTHS
,read some records, and then restore the original settings(seePasswd Functions,for an example of such a function).
NOTE: This section discusses an advancedfeature of gawk. If you are a novice awk user,you might want to skip it on the first reading.
Normally, when usingFS
, gawk defines the fields as theparts of the record that occur in between each field separator. In otherwords,FS
defines what a field is not, instead of what a fieldis. However, there are times when you really want to define the fields bywhat they are, and not by what they are not.
The most notorious such caseis so-called comma separated value (CSV) data. Many spreadsheet programs,for example, can export their data into text files, where each record isterminated with a newline, and fields are separated by commas. If onlycommas separated the data, there wouldn't be an issue. The problem comes whenone of the fields contains anembedded comma. While there is noformal standard specification for CSV data22,in such cases, most programs embed the field in double quotes. So we mighthave data like this:
Robbins,Arnold,"1234 A Pretty Street, NE",MyTown,MyState,12345-6789,USA
TheFPAT
variable offers a solution for cases like this. The value of FPAT
should be a string that provides a regular expression. This regular expression describes the contents of each field.
In the case of CSV data as presented above, each field is either “anything thatis not a comma,” or “a double quote, anything that is not a double quote, and aclosing double quote.” If written as a regular expression constant(seeRegexp),we would have /([^,]+)|("[^"]+")/
. Writing this as a string requires us to escape the double quotes, leading to:
FPAT = "([^,]+)|(\"[^\"]+\")"
Putting this to use, here is a simple program to parse the data:
BEGIN { FPAT = "([^,]+)|(\"[^\"]+\")" } { print "NF = ", NF for (i = 1; i <= NF; i++) { printf("$%d = <%s>\n", i, $i) } }
When run, we get the following:
$ gawk -f simple-csv.awk addresses.csv NF = 7 $1 = <Robbins> $2 = <Arnold> $3 = <"1234 A Pretty Street, NE"> $4 = <MyTown> $5 = <MyState> $6 = <12345-6789> $7 = <USA>
Note the embedded comma in the value of $3
.
A straightforward improvement when processing CSV data of this sortwould be to remove the quotes when they occur, with something like this:
if (substr($i, 1, 1) == "\"") { len = length($i) $i = substr($i, 2, len - 2) # Get text within the two quotes }
As with FS
, the IGNORECASE
variable (see User-modified)affects field splitting with FPAT
.
Similar to FIELDWIDTHS
, the value of PROCINFO["FS"]
will be"FPAT"
if content-based field splitting is being used.
NOTE: Some programs export CSV data that contains embedded newlines betweenthe double quotes.
gawk provides no way to deal with this. Since there is no formal specification for CSV data, there isn't muchmore to be done;the
FPAT
mechanism provides an elegant solution for the majorityof cases, and the
gawk maintainer is satisfied with that.
As written, the regexp used for FPAT
requires that each fieldhave a least one character. A straightforward modification(changing changed the first ‘+’ to ‘*’) allows fields to be empty:
FPAT = "([^,]*)|(\"[^\"]+\")"
Finally, the patsplit()
function makes the same functionalityavailable for splitting regular strings (seeString Functions).
In some databases, a single line cannot conveniently hold all theinformation in one entry. In such cases, you can use multilinerecords. The first step in doing this is to choose your data format.
One technique is to use an unusual character or string to separaterecords. For example, you could use the formfeed character (written‘\f’ inawk, as in C) to separate them, making each recorda page of the file. To do this, just set the variableRS
to"\f"
(a string containing the formfeed character). Anyother character could equally well be used, as long as it won't be partof the data in a record.
Another technique is to have blank lines separate records. By a specialdispensation, an empty string as the value ofRS
indicates thatrecords are separated by one or more blank lines. WhenRS
is setto the empty string, each record always ends at the first blank lineencountered. The next record doesn't start until the first nonblankline that follows. No matter how many blank lines appear in a row, theyall act as one record separator. (Blank lines must be completely empty; lines that contain onlywhitespace do not count.)
You can achieve the same effect as ‘RS = ""’ by assigning thestring"\n\n+"
to RS
. This regexp matches the newlineat the end of the record and one or more blank lines after the record. In addition, a regular expression always matches the longest possiblesequence when there is a choice(seeLeftmost Longest). So the next record doesn't start untilthe first nonblank line that follows—no matter how many blank linesappear in a row, they are considered one record separator.
There is an important difference between ‘RS = ""’ and‘RS = "\n\n+"’. In the first case, leading newlines in the inputdata file are ignored, and if a file ends without extra blank linesafter the last record, the final newline is removed from the record. In the second case, this special processing is not done. (d.c.)
Now that the input is separated into records, the second step is toseparate the fields in the record. One way to do this is to divide eachof the lines into fields in the normal manner. This happens by defaultas the result of a special feature. When RS
is set to the emptystring,and FS
is set to a single character,the newline character always acts as a field separator. This is in addition to whatever field separations result fromFS
.23
The original motivation for this special exception was probably to provideuseful behavior in the default case (i.e.,FS
is equalto " "
). This feature can be a problem if you really don'twant the newline character to separate fields, because there is no way toprevent it. However, you can work around this by using thesplit()
function to break up the record manually(see String Functions). If you have a single character field separator, you can work aroundthe special feature in a different way, by makingFS
into aregexp for that single character. For example, if the fieldseparator is a percent character, instead of‘FS = "%"’, use ‘FS = "[%]"’.
Another way to separate fields is toput each field on a separate line: to do this, just set thevariableFS
to the string "\n"
. (This singlecharacter separator matches a single newline.) A practical example of a data file organized this way might be a mailinglist, where each entry is separated by blank lines. Consider a mailinglist in a file named addresses, which looks like this:
Jane Doe 123 Main Street Anywhere, SE 12345-6789 John Smith 456 Tree-lined Avenue Smallville, MW 98765-4321 ...
A simple program to process this file is as follows:
# addrs.awk --- simple mailing list program # Records are separated by blank lines. # Each line is one field. BEGIN { RS = "" ; FS = "\n" } { print "Name is:", $1 print "Address is:", $2 print "City and State are:", $3 print "" }
Running the program produces the following output:
$ awk -f addrs.awk addresses -| Name is: Jane Doe -| Address is: 123 Main Street -| City and State are: Anywhere, SE 12345-6789 -| -| Name is: John Smith -| Address is: 456 Tree-lined Avenue -| City and State are: Smallville, MW 98765-4321 -| ...
See Labels Program, for a more realisticprogram that deals with address lists. The followingtablesummarizes how records are split, based on thevalue ofRS
:
RS == "\n"
RS ==
any single character
RS == ""
FS
is a single character, thenthe newline characteralways serves as a field separator, in addition to whatever value
FS
may have. Leading and trailing newlines in a file are ignored.
RS ==
regexp
In all cases,gawk sets RT
to the input text that matched thevalue specified byRS
. But if the input file ended without any text that matches RS
,then gawk sets RT
to the null string.
getline
So far we have been getting our input data fromawk's maininput stream—either the standard input (usually your terminal, sometimesthe output from another program) or from thefiles specified on the command line. Theawk language has aspecial built-in command calledgetline
thatcan be used to read input under your explicit control.
The getline
command is used in several different ways and shouldnot be used by beginners. The examples that follow the explanation of thegetline
commandinclude material that has not been covered yet. Therefore, come backand study thegetline
command after you have reviewed therest of this Web page and have a good knowledge of howawk works.
Thegetline
command returns one if it finds a record and zero ifit encounters the end of the file. If there is some error in gettinga record, such as a file that cannot be opened, thengetline
returns −1. In this case, gawk sets the variableERRNO
to a string describing the error that occurred.
In the following examples, command stands for a string value thatrepresents a shell command.
NOTE: When --sandbox is specified (see Options),reading lines from files, pipes and coprocesses is disabled.
getline
with No ArgumentsThe getline
command can be used without arguments to read inputfrom the current input file. All it does in this case is read the nextinput record and split it up into fields. This is useful if you'vefinished processing the current record, but want to do some specialprocessing on the next record right now. For example:
{ if ((t = index($0, "/*")) != 0) { # value of `tmp' will be "" if t is 1 tmp = substr($0, 1, t - 1) u = index(substr($0, t + 2), "*/") offset = t + 2 while (u == 0) { if (getline <= 0) { m = "unexpected EOF or error" m = (m ": " ERRNO) print m > "/dev/stderr" exit } u = index($0, "*/") offset = 0 } # substr() expression will be "" if */ # occurred at end of line $0 = tmp substr($0, offset + u + 2) } print $0 }
This awk program deletes C-style comments (‘/* ... */’) from the input. By replacing the ‘print $0’ with otherstatements, you could perform more complicated processing on thedecommented input, such as searching for matches of a regularexpression. (This program has a subtle problem—it does not work if onecomment ends and another begins on the same line.)
This form of the getline
command sets NF
,NR
,FNR
, and the value of $0
.
NOTE: The new value of$0
is used to testthe patterns of any subsequent rules. The original valueof$0
that triggered the rule that executedgetline
is lost. By contrast, thenext
statement reads a new recordbut immediately begins processing it normally, starting with the firstrule in the program. See Next Statement.
getline
into a VariableYou can use ‘getlinevar’ to read the next record fromawk's input into the variablevar. No other processing isdone. For example, suppose the next line is a comment or a special string,and you want to read it without triggeringany rules. This form ofgetline
allows you to read that lineand store it in a variable so that the mainread-a-line-and-check-each-rule loop ofawk never sees it. The following example swaps every two lines of input:
{ if ((getline tmp) > 0) { print tmp print $0 } else print $0 }
It takes the following list:
wan tew free phore
and produces these results:
tew wan phore free
The getline
command used in this way sets only the variablesNR
andFNR
(and of course, var). The record is notsplit into fields, so the values of the fields (including$0
) andthe value of NF
do not change.
getline
from a FileUse ‘getline < file’ to read the next record fromfile. Here file is a string-valued expression thatspecifies the file name. ‘<file’ is called a redirectionbecause it directs input to come from a different place. For example, the followingprogram reads its input record from the filesecondary.input when itencounters a first field with a value equal to 10 in the current inputfile:
{ if ($1 == 10) { getline < "secondary.input" print } else print }
Because the main input stream is not used, the values of NR
andFNR
are not changed. However, the record it reads is split into fields inthe normal manner, so the values of$0
and the other fields arechanged, resulting in a new value of NF
.
According to POSIX, ‘getline <expression’ is ambiguous ifexpression contains unparenthesized operators other than‘$’; for example, ‘getline < dir "/" file’ is ambiguousbecause the concatenation operator is not parenthesized. You shouldwrite it as ‘getline < (dir "/" file)’ if you want your programto be portable to allawk implementations.
getline
into a Variable from a FileUse ‘getlinevar < file’ to read inputfrom the filefile, and put it in the variablevar. As above, fileis a string-valued expression that specifies the file from which to read.
In this version ofgetline
, none of the built-in variables arechanged and the record is not split into fields. The only variablechanged isvar.24For example, the following program copies all the input files to theoutput, except for records that say ‘@include filename’. Such a record is replaced by the contents of the filefilename:
{ if (NF == 2 && $1 == "@include") { while ((getline line < $2) > 0) print line close($2) } else print }
Note here how the name of the extra input file is not built intothe program; it is taken directly from the data, specifically from the second field onthe ‘@include’ line.
The close()
function is called to ensure that if two identical‘@include’ lines appear in the input, the entire specified file isincluded twice. SeeClose Files And Pipes.
One deficiency of this program is that it does not process nested‘@include’ statements(i.e., ‘@include’ statements in included files)the way a true macro preprocessor would. SeeIgawk Program, for a programthat does handle nested ‘@include’ statements.
getline
from a PipeThe output of a command can also be piped into getline
, using‘command | getline’. Inthis case, the stringcommand is run as a shell command and its outputis piped into awk to be used as input. This form ofgetline
reads one record at a time from the pipe. For example, the following program copies its input to its output, except forlines that begin with ‘@execute’, which are replaced by the outputproduced by running the rest of the line as a shell command:
{ if ($1 == "@execute") { tmp = substr($0, 10) # Remove "@execute" while ((tmp | getline) > 0) print close(tmp) } else print }
Theclose()
function is called to ensure that if two identical‘@execute’ lines appear in the input, the command is run foreach one. SeeClose Files And Pipes. Given the input:
foo bar baz @execute who bletch
the program might produce:
foo bar baz arnold ttyv0 Jul 13 14:22 miriam ttyp0 Jul 13 14:23 (murphy:0) bill ttyp1 Jul 13 14:23 (murphy:0) bletch
Notice that this program ran the command who and printed the previous result. (If you try this program yourself, you will of course get different results,depending upon who is logged in on your system.)
This variation of getline
splits the record into fields, sets thevalue ofNF
, and recomputes the value of $0
. The values ofNR
andFNR
are not changed.
According to POSIX, ‘expression | getline’ is ambiguous ifexpression contains unparenthesized operators other than‘$’—for example, ‘"echo " "date" | getline’ is ambiguousbecause the concatenation operator is not parenthesized. You shouldwrite it as ‘("echo " "date") | getline’ if you want your programto be portable to all awk implementations.
NOTE: Unfortunately, gawk has not been consistent in its treatmentof a construct like ‘ "echo " "date" | getline’. Most versions, including the current version, treat it at as‘ ("echo " "date") | getline’. (This how Brian Kernighan's awk behaves.) Some versions changed and treated it as‘ "echo " ("date" | getline)’. (This is how mawk behaves.) In short, always use explicit parentheses, and then you won'thave to worry.
getline
into a Variable from a PipeWhen you use ‘command | getlinevar’, theoutput of command is sent through a pipe togetline
and into the variablevar. For example, thefollowing program reads the current date and time into the variablecurrent_time
, using thedate utility, and thenprints it:
BEGIN { "date" | getline current_time close("date") print "Report printed on " current_time }
In this version of getline
, none of the built-in variables arechanged and the record is not split into fields.
getline
from a CoprocessInput into getline
from a pipe is a one-way operation. The command that is started with ‘command | getline’ onlysends datato your awk program.
On occasion, you might want to send data to another programfor processing and then read the results back.gawk allows you to start a coprocess, with which two-waycommunications are possible. This is done with the ‘|&’operator. Typically, you write data to the coprocess first and thenread results back, as shown in the following:
print "some query" |& "db_server" "db_server" |& getline
which sends a query to db_server and then reads the results.
The values of NR
andFNR
are not changed,because the main input stream is not used. However, the record is split into fields inthe normal manner, thus changing the values of$0
, of the other fields,and of NF
.
Coprocesses are an advanced feature. They are discussed here only becausethis is the section ongetline
. See Two-way I/O,where coprocesses are discussed in more detail.
getline
into a Variable from a CoprocessWhen you use ‘command |& getlinevar’, the output fromthe coprocess command is sent through a two-way pipe togetline
and into the variable var.
In this version of getline
, none of the built-in variables arechanged and the record is not split into fields. The only variablechanged isvar.
getline
Here are some miscellaneous points about getline
thatyou should bear in mind:
getline
changes the value of $0
and NF
,awk doesnot automatically jump to the start of theprogram and start testing the new record against every pattern. However, the new record is tested against any subsequent rules. getline
without aredirection inside aBEGIN
rule. Because an unredirected getline
reads from the command-line data files, the firstgetline
commandcauses awk to set the value ofFILENAME
. Normally,FILENAME
does not have a value insideBEGIN
rules, because youhave not yet started to process the command-line data files. (d.c.) (SeeBEGIN/END,also see Auto-set.)FILENAME
with getline
(‘getline < FILENAME’)is likely to be a source forconfusion.awk opens a separate input stream from thecurrent input file. However, by not using a variable,$0
and NR
are still updated. If you're doing this, it'sprobably by accident, and you should reconsider what it is you'retrying to accomplish.getline
variants and which variables they can affect. It is worth noting that those variants which do not use redirectioncan causeFILENAME
to be updated if they causeawk to start reading a new input file. getline
Variantstable-getline-variantssummarizes the eight variants ofgetline
,listing which built-in variables are set by each one,and whether the variant is standard or agawk extension.
Variant | Effect | Standard / Extension |
---|---|---|
getline |
Sets $0 , NF , FNR , andNR |
Standard |
getline var |
Sets var, FNR , and NR |
Standard |
getline < file |
Sets $0 and NF |
Standard |
getline var < file |
Sets var | Standard |
command | getline |
Sets $0 and NF |
Standard |
command | getline var |
Sets var | Standard |
command |& getline |
Sets $0 and NF |
Extension |
command |& getline var |
Sets var | Extension |
Table 4.1: getline Variants and What They Set
According to the POSIX standard, files named on theawkcommand line must be text files. It is a fatal error if they are not. Most versions ofawk treat a directory on the command line asa fatal error.
By default, gawk produces a warning for a directory on thecommand line, but otherwise ignores it. If either of the--posixor --traditional options is given, thengawk revertsto treating a directory on the command line as a fatal error.
One of the most common programming actions is toprint, or output,some or all of the input. Use the print
statementfor simple output, and theprintf
statementfor fancier formatting. The print
statement is not limited whencomputingwhich values to print. However, with two exceptions,you cannot specify how to print them—how manycolumns, whether to use exponential notation or not, and so on. (For the exceptions, seeOutput Separators, andOFMT.) For printing with specifications, you need theprintf
statement(see Printf).
Besides basic and formatted printing, this chapteralso covers I/O redirections to files and pipes, introducesthe special file names that gawk processes internally,and discusses theclose()
built-in function.
print
StatementThe print
statement is used for producing output with simple, standardizedformatting. Specify only the strings or numbers to print, in alist separated by commas. They are output, separated by single spaces,followed by a newline. The statement looks like this:
print item1, item2, ...
The entire list of items may be optionally enclosed in parentheses. Theparentheses are necessary if any of the item expressions uses the ‘>’relational operator; otherwise it could be confused with an output redirection(see Redirection).
The items to print can be constant strings or numbers, fields of thecurrent record (such as$1
), variables, or any awkexpression. Numeric values are converted to strings and then printed.
The simple statement ‘print’ with no items is equivalent to‘print $0’: it prints the entire current record. To print a blankline, use ‘print ""’, where""
is the empty string. To print a fixed piece of text, use a string constant, such as"Don't Panic"
, as one item. If you forget to use thedouble-quote characters, your text is taken as anawkexpression, and you will probably get an error. Keep in mind that aspace is printed between any two items.
print
Statement ExamplesEach print
statement makes at least one line of output. However, itisn't limited to only one line. If an item value is a string containing anewline, the newline is output along with the rest of the string. Asingleprint
statement can make any number of lines this way.
The following is an example of printing a string that contains embedded newlines(the ‘\n’ is an escape sequence, used to represent the newlinecharacter; seeEscape Sequences):
$ awk 'BEGIN { print "line one\nline two\nline three" }' -| line one -| line two -| line three
The next example, which is run on theinventory-shipped file,prints the first two fields of each input record, with a space betweenthem:
$ awk '{ print $1, $2 }' inventory-shipped -| Jan 13 -| Feb 15 -| Mar 15 ...
A common mistake in using theprint
statement is to omit the commabetween two items. This often has the effect of making the items runtogether in the output, with no space. The reason for this is thatjuxtaposing two string expressions inawk means to concatenatethem. Here is the same program, without the comma:
$ awk '{ print $1 $2 }' inventory-shipped -| Jan13 -| Feb15 -| Mar15 ...
To someone unfamiliar with theinventory-shipped file, neitherexample's output makes much sense. A heading line at the beginningwould make it clearer. Let's add some headings to our table of months($1
) and green crates shipped ($2
). We do this using theBEGIN
pattern(see BEGIN/END)so that the headings are only printed once:
awk 'BEGIN { print "Month Crates" print "----- ------" } { print $1, $2 }' inventory-shipped
When run, the program prints the following:
Month Crates ----- ------ Jan 13 Feb 15 Mar 15 ...
The only problem, however, is that the headings and the table datadon't line up! We can fix this by printing some spaces between thetwo fields:
awk 'BEGIN { print "Month Crates" print "----- ------" } { print $1, " ", $2 }' inventory-shipped
Lining up columns this way can get prettycomplicated when there are many columns to fix. Counting spaces for twoor three columns is simple, but any more than this can take upa lot of time. This is why theprintf
statement wascreated (see Printf);one of its specialties is lining up columns of data.
NOTE: You can continue either aprintf
statement simply by putting a newline after any comma(see Statements/Lines).
As mentioned previously, aprint
statement contains a listof items separated by commas. In the output, the items are normallyseparated by single spaces. However, this doesn't need to be the case;a single space is simply the default. Any string ofcharacters may be used as the output field separator by setting thebuilt-in variable OFS
. The initial value of this variableis the string" "
—that is, a single space.
The output from an entire print
statement is called anoutput record. Eachprint
statement outputs one outputrecord, and then outputs a string called theoutput record separator(or ORS
). The initialvalue of ORS
is the string "\n"
; i.e., a newlinecharacter. Thus, each print
statement normally makes a separate line.
In order to change how output fields and records are separated, assignnew values to the variablesOFS
and ORS
. The usualplace to do this is in the BEGIN
rule(see BEGIN/END), sothat it happens before any input is processed. It can also be donewith assignments on the command line, before the names of the inputfiles, or using the-v command-line option(see Options). The following example prints the first and second fields of each inputrecord, separated by a semicolon, with a blank line added after eachnewline:
$ awk 'BEGIN { OFS = ";"; ORS = "\n\n" } > { print $1, $2 }' BBS-list -| aardvark;555-5553 -| -| alpo-net;555-3412 -| -| barfly;555-7685 ...
If the value of ORS
does not contain a newline, the program's outputruns together on a single line.
print
When printing numeric values with theprint
statement,awk internally converts the number to a string of charactersand prints that string.awk uses the sprintf()
functionto do this conversion(seeString Functions). For now, it suffices to say that thesprintf()
function accepts a format specification that tells it how to formatnumbers (or strings), and that there are a number of different ways in whichnumbers can be formatted. The different format specifications are discussedmore fully inControl Letters.
The built-in variableOFMT
contains the default format specificationthat print
uses withsprintf()
when it wants to convert anumber to a string for printing. The default value ofOFMT
is "%.6g"
. The way print
prints numbers can be changedby supplying different format specificationsas the value ofOFMT
, as shown in the following example:
$ awk 'BEGIN { > OFMT = "%.0f" # print numbers as integers (rounds) > print 17.23, 17.54 }' -| 17 18
According to the POSIX standard, awk's behavior is undefinedifOFMT
contains anything but a floating-point conversion specification. (d.c.)
printf
Statements for Fancier PrintingFor more precise control over the output format than what isprovided byprint
, use printf
. With printf
you canspecify the width to use for each item, as well as variousformatting choices for numbers (such as what output base to use, whether toprint an exponent, whether to print a sign, and how many digits to printafter the decimal point). You do this by supplying a string, calledtheformat string, that controls how and where to print the otherarguments.
printf
StatementA simpleprintf
statement looks like this:
printf format, item1, item2, ...
The entire list of arguments may optionally be enclosed in parentheses. Theparentheses are necessary if any of the item expressions use the ‘>’relational operator; otherwise, it can be confused with an output redirection(see Redirection).
The difference between printf
andprint
is the formatargument. This is an expression whose value is taken as a string; itspecifies how to output each of the other arguments. It is called theformat string.
The format string is very similar to that in the ISO C library functionprintf()
. Most offormat is text to output verbatim. Scattered among this text are format specifiers—one per item. Each format specifier says to output the next item in the argument listat that place in the format.
The printf
statement does not automatically append a newlineto its output. It outputs only what the format string specifies. So if a newline is needed, you must include one in the format string. The output separator variablesOFS
and ORS
have no effecton printf
statements. For example:
$ awk 'BEGIN { > ORS = "\nOUCH!\n"; OFS = "+" > msg = "Dont Panic!" > printf "%s\n", msg > }' -| Dont Panic!
Here, neither the ‘+’ nor the ‘OUCH’ appear inthe output message.
A format specifier starts with the character ‘%’ and ends witha format-control letter—it tells the printf
statementhow to output one item. The format-control letter specifies whatkindof value to print. The rest of the format specifier is made up ofoptionalmodifiers that control how to print the value, such asthe field width. Here is a list of the format-control letters:
%c
NOTE: The POSIX standard says the first character of a string is printed. In locales with multibyte characters, gawk attempts toconvert the leading bytes of the string into a valid wide characterand then to print the multibyte encoding of that character. Similarly, when printing a numeric value, gawk allows thevalue to be within the numeric range of values that can be heldin a wide character.Other awk versions generally restrict themselves to printingthe first byte of a string or to numeric values within the range ofa single byte (0–255).
%d
,
%i
%e
,
%E
printf "%4.3e\n", 1950
prints ‘1.950e+03’, with a total of four significant figures, three ofwhich follow the decimal point. (The ‘4.3’ represents two modifiers,discussed in the next subsection.) ‘%E’ uses ‘E’ instead of ‘e’ in the output.
%f
printf "%4.3f", 1950
prints ‘1950.000’, with a total of four significant figures, three ofwhich follow the decimal point. (The ‘4.3’ represents two modifiers,discussed in the next subsection.)
On systems supporting IEEE 754 floating point format, valuesrepresenting negativeinfinity are formatted as‘-inf’ or ‘-infinity’,and positive infinity as‘inf’ and ‘infinity’. The special “not a number” value formats as ‘-nan’ or ‘nan’.
%F
The ‘%F’ format is a POSIX extension to ISO C; not all systemssupport it. On those that don't,gawk uses ‘%f’ instead.
%g
,
%G
%o
%s
%u
%x
,
%X
%%
NOTE: When using the integer format-control letters for values that areoutside the range of the widest C integer type, gawk switches tothe ‘ %g’ format specifier. If --lint is provided on thecommand line (see Options), gawkwarns about this. Other versions of awk may print invalidvalues or do something else entirely. (d.c.)
printf
FormatsA format specification can also includemodifiers that can controlhow much of the item's value is printed, as well as how much space it gets. The modifiers come between the ‘%’ and the format-control letter. We will use the bullet symbol “•” in the following examples torepresentspaces in the output. Here are the possible modifiers, in the order inwhich they may appear:
$
printf "%s %s\n", "don't", "panic" printf "%2$s %1$s\n", "panic", "don't"
prints the famous friendly message twice.
At first glance, this feature doesn't seem to be of much use. It is in fact a gawk extension, intended for use in translatingmessages at runtime. SeePrintf Ordering,which describes how and why to use positional specifiers. For now, we will not use them.
-
printf "%-4s", "foo"
prints ‘foo•’.
+
#
0
'
$ cat thousands.awk Show source program -| BEGIN { printf "%'d\n", 1234567 } $ LC_ALL=C gawk -f thousands.awk -| 1234567 Results in "C" locale $ LC_ALL=en_US.UTF-8 gawk -f thousands.awk -| 1,234,567 Results in US English UTF locale
For more information about locales and internationalization issues,seeLocales.
NOTE: The ‘ '’ flag is a nice feature, but its use complicates things: itbecomes difficult to use it in command-line programs. For informationon appropriate quoting tricks, see Quoting.
printf "%4s", "foo"
prints ‘•foo’.
The value of width is a minimum width, not a maximum. If the itemvalue requires more thanwidth characters, it can be as wide asnecessary. Thus, the following:
printf "%4s", "foobar"
prints ‘foobar’.
Preceding the width with a minus sign causes the output to bepadded with spaces on the right, instead of on the left.
.
prec
%d
,
%i
,
%o
,
%u
,
%x
,
%X
%e
,
%E
,
%f
,
%F
%g
,
%G
%s
Thus, the following:
printf "%.4s", "foobar"
prints ‘foob’.
The C library printf
's dynamic width and preccapability (for example,"%*.*s"
) is supported. Instead ofsupplying explicit width and/orprec values in the formatstring, they are passed in the argument list. For example:
w = 5 p = 3 s = "abcdefg" printf "%*.*s\n", w, p, s
is exactly equivalent to:
s = "abcdefg" printf "%5.3s\n", s
Both programs output ‘••abc’. Earlier versions ofawk did not support this capability. If you must use such a version, you may simulate this feature by usingconcatenation to build up the format string, like so:
w = 5 p = 3 s = "abcdefg" printf "%" w "." p "s\n", s
This is not particularly easy to read but it does work.
C programmers may be used to supplying additional‘l’, ‘L’, and ‘h’modifiers inprintf
format strings. These are not valid in awk. Mostawk implementations silently ignore them. If--lint is provided on the command line(seeOptions),gawk warns about their use. If--posix is supplied,their use is a fatal error.
printf
The following simple example showshow to use printf
to make an aligned table:
awk '{ printf "%-10s %s\n", $1, $2 }' BBS-list
This commandprints the names of the bulletin boards ($1
) in the fileBBS-list as a string of 10 characters that are left-justified. It alsoprints the phone numbers ($2
) next on the line. Thisproduces an aligned two-column table of names and phone numbers,as shown here:
$ awk '{ printf "%-10s %s\n", $1, $2 }' BBS-list -| aardvark 555-5553 -| alpo-net 555-3412 -| barfly 555-7685 -| bites 555-1675 -| camelot 555-0542 -| core 555-2912 -| fooey 555-1234 -| foot 555-6699 -| macfoo 555-6480 -| sdace 555-3430 -| sabafoo 555-2127
In this case, the phone numbers had to be printed as strings becausethe numbers are separated by a dash. Printing the phone numbers asnumbers would have produced just the first three digits: ‘555’. This would have been pretty confusing.
It wasn't necessary to specify a width for the phone numbers becausethey are last on their lines. They don't need to have spacesafter them.
The table could be made to look even nicer by adding headings to thetops of the columns. This is done using theBEGIN
pattern(see BEGIN/END)so that the headers are only printed once, at the beginning oftheawk program:
awk 'BEGIN { print "Name Number" print "---- ------" } { printf "%-10s %s\n", $1, $2 }' BBS-list
The above example mixes print
and printf
statements inthe same program. Using justprintf
statements can produce thesame results:
awk 'BEGIN { printf "%-10s %s\n", "Name", "Number" printf "%-10s %s\n", "----", "------" } { printf "%-10s %s\n", $1, $2 }' BBS-list
Printing each column heading with the same format specificationused for the column elements ensures that the headingsare aligned just like the columns.
The fact that the same format specification is used three times can beemphasized by storing it in a variable, like this:
awk 'BEGIN { format = "%-10s %s\n" printf format, "Name", "Number" printf format, "----", "------" } { printf format, $1, $2 }' BBS-list
At this point, it would be a worthwhile exercise to use theprintf
statement to line up the headings and table data for theinventory-shipped example that was covered earlier in the sectionon theprint
statement(see Print).
print
and printf
So far, the output from print
and printf
has goneto the standardoutput, usually the screen. Bothprint
and printf
canalso send their output to other places. This is calledredirection.
NOTE: When --sandbox is specified (see Options),redirecting output to files and pipes is disabled.
A redirection appears after the print
or printf
statement. Redirections inawk are written just like redirections in shellcommands, except that they are written inside theawk program.
There are four forms of output redirection: output to a file, outputappended to a file, output through a pipe to another command, and outputto a coprocess. They are all shown for theprint
statement,but they work identically for printf
:
print
items
>
output-file
When this type of redirection is used, the output-file is erasedbefore the first output is written to it. Subsequent writes to the sameoutput-file do not eraseoutput-file, but append to it. (This is different from how you use redirections in shell scripts.) Ifoutput-file does not exist, it is created. For example, hereis how an awk program can write a list of BBS names to onefile namedname-list, and a list of phone numbers to another filenamedphone-list:
$ awk '{ print $2 > "phone-list" > print $1 > "name-list" }' BBS-list $ cat phone-list -| 555-5553 -| 555-3412 ... $ cat name-list -| aardvark -| alpo-net ...
Each output file contains one name or number per line.
print
items
>>
output-file
print
items
|
command
The redirection argument command is actually an awkexpression. Its value is converted to a string whose contents givethe shell command to be run. For example, the following produces twofiles, one unsorted list of BBS names, and one list sorted in reversealphabetical order:
awk '{ print $1 > "names.unsorted" command = "sort -r > names.sorted" print $1 | command }' BBS-list
The unsorted list is written with an ordinary redirection, whilethe sorted list is written by piping through thesort utility.
The next example uses redirection to mail a message to the mailinglist ‘bug-system’. This might be useful when trouble is encounteredin anawk script run periodically for system maintenance:
report = "mail bug-system" print "Awk script failed:", $0 | report m = ("at record number " FNR " of " FILENAME) print m | report close(report)
The message is built using string concatenation and saved in the variablem
. It's then sent down the pipeline to themail program. (The parentheses group the items to concatenate—seeConcatenation.)
The close()
function is called here because it's a good idea to closethe pipe as soon as all the intended output has been sent to it. SeeClose Files And Pipes,for more information.
This example also illustrates the use of a variable to representa file orcommand—it is not necessary to alwaysuse a string constant. Using a variable is generally a good idea,because (if you mean to refer to that same file or command)awk requires that the string value be spelled identicallyevery time.
print
items
|&
command
getline
. Thus command is a coprocess, which works together with,but subsidiary to, the
awk program.
This feature is a gawk extension, and is not available inPOSIXawk. See Getline/Coprocess,for a brief discussion. See Two-way I/O,for a more complete discussion.
Redirecting output using ‘>’, ‘>>’, ‘|’, or ‘|&’asks the system to open a file, pipe, or coprocess only if the particularfile or command you specify has not already been writtento by your program or if it has been closed since it was last written to.
It is a common error to use ‘>’ redirection for the firstprint
to a file, and then to use ‘>>’ for subsequent output:
# clear the file print "Don't panic" > "guide.txt" ... # append print "Avoid improbability generators" >> "guide.txt"
This is indeed how redirections must be used from the shell. But inawk, it isn't necessary. In this kind of case, a program shoulduse ‘>’ for all theprint
statements, since the output fileis only opened once. (It happens that if you mix ‘>’ and ‘>>’that output is produced in the expected order. However, mixing the operatorsfor the same file is definitely poor style, and is confusing to readersof your program.)
As mentioned earlier(see Getline Notes),manyManyolderawk implementations limit the number of pipelines that anawkprogram may have open to just one! Ingawk, there is no such limit. gawk allows a program toopen as many pipelines as the underlying operating system permits.
A particularly powerful way to use redirection is to build command linesand pipe them into the shell,sh. For example, suppose youhave a list of files brought over from a system where all the file namesare stored in uppercase, and you wish to rename them to have names inall lowercase. The following program is both simple and efficient:
{ printf("mv %s %s\n", $0, tolower($0)) | "sh" } END { close("sh") }
The tolower()
function returns its argument string with alluppercase characters converted to lowercase(seeString Functions). The program builds up a list of command lines,using themv utility to rename the files. It then sends the list to the shell for execution.
gawk provides a number of special file names that it interpretsinternally. These file names provide access to standard file descriptorsand TCP/IP networking.
Running programs conventionally have three input and output streamsalready available to them for reading and writing. These are known asthestandard input, standard output, and standard erroroutput. These streams are, by default, connected to your keyboard and screen, butthey are often redirected with the shell, via the ‘<’, ‘<<’,‘>’, ‘>>’, ‘>&’, and ‘|’ operators. Standard erroris typically used for writing error messages; the reason there are two separatestreams, standard output and standard error, is so that they can beredirected separately.
In other implementations ofawk, the only way to write an errormessage to standard error in anawk program is as follows:
print "Serious error detected!" | "cat 1>&2"
This works by opening a pipeline to a shell command that can access thestandard error stream that it inherits from theawk process. This is far from elegant, and it is also inefficient, because it requires aseparate process. So people writingawk programs oftendon't do this. Instead, they send the error messages to thescreen, like this:
print "Serious error detected!" > "/dev/tty"
(/dev/tty is a special file supplied by the operating systemthat is connected to your keyboard and screen. It represents the“terminal,”25 which on modern systems is a keyboardand screen, not a serial console.) This usually has the same effect but not always: although thestandard error stream is usually the screen, it can be redirected; whenthat happens, writing to the screen is not correct. In fact, ifawk is run from a background job, it may not have aterminal at all. Then opening/dev/tty fails.
gawk provides special file names for accessing the three standardstreams. (c.e.). It also provides syntax for accessingany other inherited open files. If the file name matchesone of these special names whengawk redirects input or output,then it directly uses the stream that the file name stands for. These special file names work for all operating systems thatgawkhas been ported to, not just those that are POSIX-compliant:
The file names /dev/stdin, /dev/stdout, and/dev/stderrare aliases for /dev/fd/0,/dev/fd/1, and /dev/fd/2,respectively. However, they are more self-explanatory. The proper way to write an error message in agawk programis to use /dev/stderr, like this:
print "Serious error detected!" > "/dev/stderr"
Note the use of quotes around the file name. Like any other redirection, the value must be a string. It is a common error to omit the quotes, which leadsto confusing results.
Finally, using the close()
function on a file name of theform "/dev/fd/
N"
, for file descriptor numbersabove two, will actually close the given file descriptor.
The /dev/stdin, /dev/stdout, and/dev/stderrspecial files are also recognized internally by several otherversions ofawk.
gawk programscan open a two-wayTCP/IP connection, acting as either a client or a server. This is done using a special file name of the form:
/net-type/protocol/local-port/remote-host/remote-port
The net-type is one of ‘inet’, ‘inet4’ or ‘inet6’. Theprotocol is one of ‘tcp’ or ‘udp’,and the other fields represent the other essential pieces of informationfor making a networking connection. These file names are used with the ‘|&’ operator for communicatingwith a coprocess(seeTwo-way I/O). This is an advanced feature, mentioned here only for completeness. Full discussion is delayed untilTCP/IP Networking.
Here is a list of things to bear in mind when using thespecial file names thatgawk provides:
dup()
'ed from file descriptor 4. Most ofthe time this does not matter; however, it is important tonotclose any of the files related to file descriptors 0, 1, and 2. Doing so results in unpredictable behavior.If the same file name or the same shell command is used with getline
more than once during the execution of anawk program(see Getline),the file is opened (or the command is executed) the first time only. At that time, the first record of input is read from that file or command. The next time the same file or command is used with getline
,another record is read from it, and so on.
Similarly, when a file or pipe is opened for output, awk remembersthe file name or command associated with it, and subsequentwrites to the same file or command are appended to the previous writes. The file or pipe stays open until awk exits.
This implies that special steps are necessary in order to read the samefile again from the beginning, or to rerun a shell command (rather thanreading more output from the same command). The close()
functionmakes these things possible:
close(filename)
or:
close(command)
The argument filename or command can be any expression. Itsvalue mustexactly match the string that was used to open the file orstart the command (spaces and other “irrelevant” charactersincluded). For example, if you open a pipe with this:
"sort -r names" | getline foo
then you must close it with this:
close("sort -r names")
Once this function call is executed, the next getline
from thatfile or command, or the nextprint
or printf
to thatfile or command, reopens the file or reruns the command. Because the expression that you use to close a file or pipeline mustexactly match the expression used to open the file or run the command,it is good practice to use a variable to store the file name or command. The previous example becomes the following:
sortcom = "sort -r names" sortcom | getline foo ... close(sortcom)
This helps avoid hard-to-find typographical errors in your awkprograms. Here are some of the reasons for closing an output file:
getline
. For example, suppose a program pipes output to the mail program. If it outputs several lines redirected to this pipe without closingit, they make a single message of several lines. By contrast, if theprogram closes the pipe after each line of output, then each line makesa separate message.
If you use more files than the system allows you to have open,gawk attempts to multiplex the available open files amongyour data files.gawk's ability to do this depends upon thefacilities of your operating system, so it may not always work. It istherefore both good practice and good portability advice to alwaysuseclose()
on your files when you are done with them. In fact, if you are using a lot of pipes, it is essential thatyou close commands when done. For example, consider something like this:
{ ... command = ("grep " $1 " /some/file | my_prog -q " $3) while ((command | getline) > 0) { process output of command } # need close(command) here }
This example creates a new pipeline based on data in each record. Without the call toclose()
indicated in the comment, awkcreates child processes to run the commands, until it eventuallyruns out of file descriptors for more pipelines.
Even though each command has finished (as indicated by the end-of-filereturn status fromgetline
), the child process is notterminated;26more importantly, the file descriptor for the pipeis not closed and released untilclose()
is called orawk exits.
close()
will silently do nothing if given an argument thatdoes not represent a file, pipe or coprocess that was opened witha redirection.
Note also that ‘close(FILENAME)’ has no“magic” effects on the implicit loop that reads through thefiles named on the command line. It is, more likely, a closeof a file that was never opened, soawk silentlydoes nothing.
When using the ‘|&’ operator to communicate with a coprocess,it is occasionally useful to be able to close one end of the two-waypipe without closing the other. This is done by supplying a second argument toclose()
. As in any other call to close()
,the first argument is the name of the command or special file usedto start the coprocess. The second argument should be a string, with either of the values"to"
or"from"
. Case does not matter. As this is an advanced feature, a more complete discussion isdelayed untilTwo-way I/O,which discusses it in more detail and gives an example.
close()
's Return ValueIn many versions of Unix awk, the close()
functionis actually a statement. It is a syntax error to try and use the returnvalue fromclose()
:(d.c.)
command = "..." command | getline info retval = close(command) # syntax error in many Unix awks
gawk treatsclose()
as a function. The return value is −1 if the argument names somethingthat was never opened with a redirection, or if there isa system problem closing the file or process. In these cases,gawk sets the built-in variableERRNO
to a string describing the problem.
In gawk,when closing a pipe or coprocess (input or output),the return value is the exit status of the command.27Otherwise, it is the return value from the system's close()
orfclose()
C functions when closing input or outputfiles, respectively. This value is zero if the close succeeds, or −1 ifit fails.
The POSIX standard is very vague; it says that close()
returns zero on success and nonzero otherwise. In general,different implementations vary in what they report when closingpipes; thus the return value cannot be used portably. (d.c.) In POSIX mode (see Options), gawk just returns zerowhen closing a pipe.
Expressions are the basic building blocks ofawk patternsand actions. An expression evaluates to a value that you can print, test,or pass to a function. Additionally, an expressioncan assign a new value to a variable or a field by using an assignment operator.
An expression can serve as a pattern or action statement on its own. Most other kinds ofstatements contain one or more expressions that specify the data on which tooperate. As in other languages, expressions inawk includevariables, array references, constants, and function calls, as well ascombinations of these with various operators.
Expressions are built up from values and the operations performedupon them. This section describes the elementary objectswhich provide the values used in expressions.
The simplest type of expression is theconstant, which always hasthe same value. There are three types of constants: numeric,string, and regular expression.
Each is used in the appropriate context when you need a datavalue that isn't going to change. Numeric constants canhave different forms, but are stored identically internally.
A numeric constant stands for a number. This number can be aninteger, a decimal fraction, or a number in scientific (exponential)notation.28Here are some examples of numeric constants that allhave the same value:
105 1.05e+2 1050e-1
A string constant consists of a sequence of characters enclosed indouble-quotation marks. For example:
"parrot"
represents the string whose contents are ‘parrot’. Strings ingawk can be of any length, and they can contain any of the possibleeight-bit ASCII characters including ASCIInul (character code zero). Other awkimplementations may have difficulty with some character codes.
Inawk, all numbers are in decimal; i.e., base 10. Many otherprogramming languages allow you to specify numbers in other bases, oftenoctal (base 8) and hexadecimal (base 16). In octal, the numbers go 0, 1, 2, 3, 4, 5, 6, 7, 10, 11, 12, etc. Just as ‘11’, in decimal, is 1 times 10 plus 1, so‘11’, in octal, is 1 times 8, plus 1. This equals 9 in decimal. In hexadecimal, there are 16 digits. Since the everyday decimalnumber system only has ten digits (‘0’–‘9’), the letters‘a’ through ‘f’ are used to represent the rest. (Case in the letters is usually irrelevant; hexadecimal ‘a’ and ‘A’have the same value.) Thus, ‘11’, inhexadecimal, is 1 times 16 plus 1, which equals 17 in decimal.
Just by looking at plain ‘11’, you can't tell what base it's in. So, in C, C++, and other languages derived from C,there is a special notation to signify the base. Octal numbers start with a leading ‘0’,and hexadecimal numbers start with a leading ‘0x’ or ‘0X’:
11
011
0x11
This example shows the difference:
$ gawk 'BEGIN { printf "%d, %d, %d\n", 011, 11, 0x11 }' -| 9, 11, 17
Being able to use octal and hexadecimal constants in your programs is mostuseful when working with data that cannot be represented conveniently ascharacters or as regular numbers, such as binary data of various sorts.
gawk allows the use of octal and hexadecimalconstants in your program text. However, such numbers in the input dataare not treated differently; doing so by default would break oldprograms. (If you really need to do this, use the--non-decimal-datacommand-line option;seeNondecimal Data.) If you have octal or hexadecimal data,you can use thestrtonum()
function(see String Functions)to convert the data into a number. Most of the time, you will want to use octal or hexadecimal constantswhen working with the built-in bit manipulation functions;seeBitwise Functions,for more information.
Unlike some early C implementations, ‘8’ and ‘9’ are not validin octal constants; e.g.,gawk treats ‘018’ as decimal 18:
$ gawk 'BEGIN { print "021 is", 021 ; print 018 }' -| 021 is 17 -| 18
Octal and hexadecimal source code constants are agawk extension. If gawk is in compatibility mode(seeOptions),they are not available.
Once a numeric constant hasbeen converted internally into a number,gawk no longer rememberswhat the original form of the constant was; the internal value isalways used. This has particular consequences for conversion ofnumbers to strings:
$ gawk 'BEGIN { printf "0x11 is <%s>\n", 0x11 }' -| 0x11 is <17>
A regexp constant is a regular expression description enclosed inslashes, such as /^beginning and end$/
. Most regexps used inawk programs are constant, but the ‘~’ and ‘!~’matching operators can also match computed or dynamic regexps(which are just ordinary strings or variables that contain a regexp).
When used on the righthand side of the ‘~’ or ‘!~’operators, a regexp constant merely stands for the regexp that is to bematched. However, regexp constants (such as /foo/
) may be used like simple expressions. When aregexp constant appears by itself, it has the same meaning as if it appearedin a pattern, i.e., ‘($0 ~ /foo/)’(d.c.) See Expression Patterns. This means that the following two code segments:
if ($0 ~ /barfly/ || $0 ~ /camelot/) print "found"
and:
if (/barfly/ || /camelot/) print "found"
are exactly equivalent. One rather bizarre consequence of this rule is that the followingBoolean expression is valid, but does not do what the user probablyintended:
# Note that /foo/ is on the left of the ~ if (/foo/ ~ $1) print "found foo"
This code is “obviously” testing$1
for a match against the regexp/foo/
. But in fact, the expression ‘/foo/ ~ $1’ really means‘($0 ~ /foo/) ~ $1’. In other words, first match the input recordagainst the regexp /foo/
. The result is either zero or one,depending upon the success or failure of the match. That resultis then matched against the first field in the record. Because it is unlikely that you would ever really want to make this kind oftest,gawk issues a warning when it sees this construct ina program. Another consequence of this rule is that the assignment statement:
matches = /foo/
assigns either zero or one to the variable matches
, dependingupon the contents of the current input record.
Constant regular expressions are also used as the first argument forthe gensub()
,sub()
, and gsub()
functions, as thesecond argument of thematch()
function,and as the third argument of the patsplit()
function(seeString Functions). Modern implementations of awk, including gawk, allowthe third argument ofsplit()
to be a regexp constant, but someolder implementations do not. (d.c.) This can lead to confusion when attempting to use regexp constantsas arguments to user-defined functions(seeUser-defined). For example:
function mysub(pat, repl, str, global) { if (global) gsub(pat, repl, str) else sub(pat, repl, str) return str } { ... text = "hi! hi yourself!" mysub(/hi/, "howdy", text, 1) ... }
In this example, the programmer wants to pass a regexp constant to theuser-defined functionmysub
, which in turn passes it on toeither sub()
or gsub()
. However, what really happens is thatthe pat
parameter is either one or zero, depending upon whetheror not$0
matches /hi/
. gawk issues a warning when it sees a regexp constant used asa parameter to a user-defined function, since passing a truth value inthis way is probably not what was intended.
Variables are ways of storing values at one point in your program foruse later in another part of your program. They can be manipulatedentirely within the program text, and they can also be assigned valueson the awk command line.
Variables let you give names to values and refer to them later. Variableshave already been used in many of the examples. The name of a variablemust be a sequence of letters, digits, or underscores, and it may not beginwith a digit. Case is significant in variable names; a
and A
are distinct variables.
A variable name is a valid expression by itself; it represents thevariable's current value. Variables are given new values withassignment operators,increment operators, anddecrement operators. See Assignment Ops. In addition, the sub()
and gsub()
functions canchange a variable's value, and thematch()
, patsplit()
and split()
functions can change the contents of theirarray parameters. SeeString Functions.
A few variables have special built-in meanings, such asFS
(thefield separator), and NF
(the number of fields in the current inputrecord). SeeBuilt-in Variables, for a list of the built-in variables. These built-in variables can be used and assigned just like all othervariables, but their values are also used or changed automatically byawk. All built-in variables' names are entirely uppercase.
Variables in awk can be assigned either numeric or string values. The kind of value a variable holds can change over the life of a program. By default, variables are initialized to the empty string, whichis zero if converted to a number. There is no need to explicitly“initialize” a variable inawk,which is what you would do in C and in most other traditional languages.
Anyawk variable can be set by including a variable assignmentamong the arguments on the command line when awk is invoked(seeOther Arguments). Such an assignment has the following form:
variable=text
With it, a variable is set either at the beginning of theawk run or in between input files. When the assignment is preceded with the -v option,as in the following:
-v variable=text
the variable is set at the very beginning, even before theBEGIN
rules execute. The-v option and its assignmentmust precede all the file name arguments, as well as the program text. (SeeOptions, for more information aboutthe -v option.) Otherwise, the variable assignment is performed at a time determined byits position among the input file arguments—after the processing of thepreceding input file argument. For example:
awk '{ print $n }' n=4 inventory-shipped n=2 BBS-list
prints the value of field number n
for all input records. Beforethe first file is read, the command line sets the variablen
equal to four. This causes the fourth field to be printed in lines frominventory-shipped. After the first file has finished,but before the second file is started,n
is set to two, so that thesecond field is printed in lines from BBS-list:
$ awk '{ print $n }' n=4 inventory-shipped n=2 BBS-list -| 15 -| 24 ... -| 555-5553 -| 555-3412 ...
Command-line arguments are made available for explicit examination bytheawk program in the ARGV
array(seeARGC and ARGV). awk processes the values of command-line assignments for escapesequences(seeEscape Sequences). (d.c.)
Strings are converted to numbers and numbers are converted to strings, if the contextof the awk program demands it. For example, if the value ofeitherfoo
or bar
in the expression ‘foo + bar’happens to be a string, it is converted to a number before the additionis performed. If numeric values appear in string concatenation, theyare converted to strings. Consider the following:
two = 2; three = 3 print (two three) + 4
This prints the (numeric) value 27. The numeric values ofthe variablestwo
and three
are converted to strings andconcatenated together. The resulting string is converted back to thenumber 23, to which 4 is then added.
If, for some reason, you need to force a number to be converted to astring, concatenate that number with the empty string,""
. To force a string to be converted to a number, add zero to that string. A string is converted to a number by interpreting any numeric prefixof the string as numerals:"2.5"
converts to 2.5,"1e3"
converts to 1000, and "25fix"
has a numeric value of 25. Strings that can't be interpreted as valid numbers convert to zero.
The exact manner in which numbers are converted into strings is controlledby theawk built-in variable CONVFMT
(seeBuilt-in Variables). Numbers are converted using thesprintf()
functionwith CONVFMT
as the formatspecifier(seeString Functions).
CONVFMT
's default value is "%.6g"
, which prints a value withat most six significant digits. For some applications, you might want tochange it to specify more precision. On most modern machines,17 digits is usually enough to capture a floating-point number'svalue exactly.29
Strange results can occur if you setCONVFMT
to a string that doesn'ttell sprintf()
how to format floating-point numbers in a useful way. For example, if you forget the ‘%’ in the format,awk convertsall numbers to the same constant string.
As a special case, if a number is an integer, then the result of convertingit to a string isalways an integer, no matter what the value ofCONVFMT
may be. Given the following code fragment:
CONVFMT = "%2.2f" a = 12 b = a ""
b
has the value "12"
, not "12.00"
. (d.c.)
Prior to the POSIX standard, awk used the valueofOFMT
for converting numbers to strings. OFMT
specifies the output format to use when printing numbers withprint
. CONVFMT
was introduced in order to separate the semantics ofconversion from the semantics of printing. BothCONVFMT
andOFMT
have the same default value: "%.6g"
. In the vast majorityof cases, oldawk programs do not change their behavior. However, these semantics forOFMT
are something to keep in mind if you mustport your new-style program to older implementations ofawk. We recommendthat instead of changing your programs, just portgawk itself. See Print,for more information on theprint
statement.
And, once again, where you are can matter when it comes to convertingbetween numbers and strings. InLocales, we mentioned thatthe local character set and language (the locale) can affect howgawk matches characters. The locale also affects numericformats. In particular, forawk programs, it affects thedecimal point character. The"C"
locale, and most English-languagelocales, use the period character (‘.’) as the decimal point. However, many (if not most) European and non-English locales use the comma(‘,’) as the decimal point character.
The POSIX standard says that awk always uses the period as the decimalpoint when reading theawk program source code, and for command-linevariable assignments (seeOther Arguments). However, when interpreting input data, forprint
and printf
output,and for number to string conversion, the local decimal point character is used. Here are some examples indicating the difference in behavior,on a GNU/Linux system:
$ gawk 'BEGIN { printf "%g\n", 3.1415927 }' -| 3.14159 $ LC_ALL=en_DK gawk 'BEGIN { printf "%g\n", 3.1415927 }' -| 3,14159 $ echo 4,321 | gawk '{ print $1 + 1 }' -| 5 $ echo 4,321 | LC_ALL=en_DK gawk '{ print $1 + 1 }' -| 5,321
The ‘en_DK’ locale is for English in Denmark, where the comma acts asthe decimal point separator. In the normal"C"
locale, gawktreats ‘4,321’ as ‘4’, while in the Danish locale, it's treatedas the full number, 4.321.
Some earlier versions of gawk fully complied with this aspectof the standard. However, many users in non-English locales complainedabout this behavior, since their data used a period as the decimalpoint, so the default behavior was restored to use a period as thedecimal point character. You can use the--use-lc-numericoption (see Options) to force gawk to use the locale'sdecimal point character. (gawk also uses the locale's decimalpoint character when in POSIX mode, either via--posix, or thePOSIXLY_CORRECT environment variable.)
table-locale-affects describes the cases in which the locale's decimalpoint character is used and when a period is used. Some of thesefeatures have not been described yet.
Feature | Default | --posix or --use-lc-numeric |
---|---|---|
%'g |
Use locale | Use locale |
%g |
Use period | Use locale |
Input | Use period | Use locale |
strtonum() |
Use period | Use locale |
Table 6.1: Locale Decimal Point versus A Period
Finally, modern day formal standards and IEEE standard floating pointrepresentation can have an unusual but important effect on the waygawk converts some special string values to numbers. The detailsare presented in POSIX Floating Point Problems.
This section introduces the operators which make useof the values provided by constants and variables.
The awk language uses the common arithmetic operators whenevaluating expressions. All of these arithmetic operators follow normalprecedence rules and work as you would expect them to.
The following example uses a file named grades, which containsa list of student names as well as three test scores per student (it'sa small class):
Pat 100 97 58 Sandy 84 72 93 Chris 72 92 89
This program takes the file grades and prints the averageof the scores:
$ awk '{ sum = $2 + $3 + $4 ; avg = sum / 3 > print $1, avg }' grades -| Pat 85 -| Sandy 83 -| Chris 84.3333
The following list provides the arithmetic operators in awk, in order fromthe highest precedence to the lowest:
-
x
+
x
^
y
**
y
*
y
/
y
%
y
+
y
-
y
Unary plus and minus have the same precedence,the multiplication operators all have the same precedence, andaddition and subtraction have the same precedence.
When computing the remainder of ‘x %y’,the quotient is rounded toward zero to an integer andmultiplied byy. This result is subtracted from x;this operation is sometimes known as “trunc-mod.” The followingrelation always holds:
b * int(a / b) + (a % b) == a
One possibly undesirable effect of this definition of remainder is thatx %
y is negative if x is negative. Thus:
-17 % 8 = -1
In other awk implementations, the signedness of the remaindermay be machine-dependent.
NOTE: The POSIX standard only specifies the use of ‘ ^’for exponentiation. For maximum portability, do not use the ‘ **’ operator.
It seemed like a good idea at the time.
Brian Kernighan
There is only one string operation: concatenation. It does not have aspecific operator to represent it. Instead, concatenation is performed bywriting expressions next to one another, with no operator. For example:
$ awk '{ print "Field number one: " $1 }' BBS-list -| Field number one: aardvark -| Field number one: alpo-net ...
Without the space in the string constant after the ‘:’, the lineruns together. For example:
$ awk '{ print "Field number one:" $1 }' BBS-list -| Field number one:aardvark -| Field number one:alpo-net ...
Because string concatenation does not have an explicit operator, it isoften necessary to insure that it happens at the right time by usingparentheses to enclose the items to concatenate. For example,you might expect that thefollowing code fragment concatenates file
andname
:
file = "file" name = "name" print "something meaningful" > file name
This produces a syntax error with some versions of Unixawk.30It is necessary to use the following:
print "something meaningful" > (file name)
Parentheses should be used around concatenation in all but themost common contexts, such as on the righthand side of ‘=’. Be careful about the kinds of expressions used in string concatenation. In particular, the order of evaluation of expressions used for concatenationis undefined in theawk language. Consider this example:
BEGIN { a = "don't" print (a " " (a = "panic")) }
It is not defined whether the assignment to a
happensbefore or after the value ofa
is retrieved for producing theconcatenated value. The result could be either ‘don't panic’,or ‘panic panic’.
The precedence of concatenation, when mixed with other operators, is oftencounter-intuitive. Consider this example:
$ awk 'BEGIN { print -12 " " -24 }' -| -12-24
This “obviously” is concatenating −12, a space, and −24. But where did the space disappear to? The answer lies in the combination of operator precedences andawk's automatic conversion rules. To get the desired result,write the program this way:
$ awk 'BEGIN { print -12 " " (-24) }' -| -12 -24
This forces awk to treat the ‘-’ on the ‘-24’ as unary. Otherwise, it's parsed as follows:
−12 (" "
− 24)
⇒ −12 (0 − 24)
⇒ −12 (−24)
⇒ −12−24
As mentioned earlier,when doing concatenation, parenthesize. Otherwise,you're never quite sure what you'll get.
Anassignment is an expression that stores a (usually different)value into a variable. For example, let's assign the value one to the variablez
:
z = 1
After this expression is executed, the variable z
has the value one. Whatever old valuez
had before the assignment is forgotten.
Assignments can also store string values. For example, thefollowing storesthe value"this food is good"
in the variable message
:
thing = "food" predicate = "good" message = "this " thing " is " predicate
This also illustrates string concatenation. The ‘=’ sign is called anassignment operator. It is thesimplest assignment operator because the value of the righthandoperand is stored unchanged. Most operators (addition, concatenation, and so on) have no effectexcept to compute a value. If the value isn't used, there's no reason touse the operator. An assignment operator is different; it doesproduce a value, but even if you ignore it, the assignment stillmakes itself felt through the alteration of the variable. We call thisaside effect.
The lefthand operand of an assignment need not be a variable(see Variables); it can also be a field(seeChanging Fields) oran array element (see Arrays). These are all called lvalues,which means they can appear on the lefthand side of an assignment operator. The righthand operand may be any expression; it produces the new valuethat the assignment stores in the specified variable, field, or arrayelement. (Such values are called rvalues.)
It is important to note that variables donot have permanent types. A variable's type is simply the type of whatever value it happensto hold at the moment. In the following program fragment, the variablefoo
has a numeric value at first, and a string value later on:
foo = 1 print foo foo = "bar" print foo
When the second assignment gives foo
a string value, the fact thatit previously had a numeric value is forgotten.
String values that do not begin with a digit have a numeric value ofzero. After executing the following code, the value offoo
is five:
foo = "a string" foo = foo + 5
NOTE: Using a variable as a number and then later as a stringcan be confusing and is poor programming style. The previous two examplesillustrate how awk works, not how you should write yourprograms!
An assignment is an expression, so it has a value—the same value thatis assigned. Thus, ‘z = 1’ is an expression with the value one. One consequence of this is that you can write multiple assignments together,such as:
x = y = z = 5
This example stores the value five in all three variables(x
,y
, and z
). It does so because thevalue of ‘z = 5’, which is five, is stored intoy
and thenthe value of ‘y = z = 5’, which is five, is stored intox
.
Assignments may be used anywhere an expression is called for. Forexample, it is valid to write ‘x != (y = 1)’ to sety
to one,and then test whether x
equals one. But this style tends to makeprograms hard to read; such nesting of assignments should be avoided,except perhaps in a one-shot program.
Aside from ‘=’, there are several other assignment operators thatdo arithmetic with the old value of the variable. For example, theoperator ‘+=’ computes a new value by adding the righthand valueto the old value of the variable. Thus, the following assignment addsfive to the value of foo
:
foo += 5
This is equivalent to the following:
foo = foo + 5
Use whichever makes the meaning of your program clearer.
There are situations where using ‘+=’ (or any assignment operator)isnot the same as simply repeating the lefthand operand in therighthand expression. For example:
# Thanks to Pat Rankin for this example BEGIN { foo[rand()] += 5 for (x in foo) print x, foo[x] bar[rand()] = bar[rand()] + 5 for (x in bar) print x, bar[x] }
The indices ofbar
are practically guaranteed to be different, becauserand()
returns different values each time it is called. (Arrays and therand()
function haven't been covered yet. See Arrays,and seeNumeric Functions, for more information). This example illustrates an important fact about assignmentoperators: the lefthand expression is only evaluatedonce. It is up to the implementation as to which expression is evaluatedfirst, the lefthand or the righthand. Consider this example:
i = 1 a[i += 2] = i + 1
The value of a[3]
could be either two or four.
table-assign-ops lists the arithmetic assignment operators. In eachcase, the righthand operand is an expression whose value is convertedto a number.
Operator | Effect |
---|---|
lvalue += increment |
Adds increment to the value of lvalue. |
lvalue -= decrement |
Subtracts decrement from the value of lvalue. |
lvalue *= coefficient |
Multiplies the value of lvalue by coefficient. |
lvalue /= divisor |
Divides the value of lvalue by divisor. |
lvalue %= modulus |
Sets lvalue to its remainder by modulus. |
lvalue ^= power |
|
lvalue **= power |
Raises lvalue to the power power. (c.e.) |
Table 6.2: Arithmetic Assignment Operators
NOTE: Only the ‘ ^=’ operator is specified by POSIX. For maximum portability, do not use the ‘ **=’ operator.
<!-- derived from email from "Nelson H. F. Beebe" -->
There is a syntactic ambiguity between the /=
assignmentoperator and regexp constants whose first character is an ‘=’. (d.c.) This is most notable in commercialawk versions. For example:
$ awk /==/ /dev/null error--> awk: syntax error at source line 1 error--> context is error--> >>> /= <<< error--> awk: bailing out at source line 1
A workaround is:
awk '/[=]=/' /dev/null
gawk does not have this problem,nor do the otherfreely available versions described inOther Versions.
Increment anddecrement operators increase or decrease the value ofa variable by one. An assignment operator can do the same thing, sothe increment operators add no power to theawk language; however, theyare convenient abbreviations for very common operations.
The operator used for adding one is written ‘++’. It can be used to incrementa variable either before or after taking its value. To pre-increment a variablev
, write ‘++v’. This addsone to the value ofv
—that new value is also the value of theexpression. (The assignment expression ‘v += 1’ is completelyequivalent.) Writing the ‘++’ after the variable specifies post-increment. Thisincrements the variable value just the same; the difference is that thevalue of the increment expression itself is the variable'soldvalue. Thus, if foo
has the value four, then the expression ‘foo++’has the value four, but it changes the value offoo
to five. In other words, the operator returns the old value of the variable,but with the side effect of incrementing it.
The post-increment ‘foo++’ is nearly the same as writing ‘(foo+= 1) - 1’. It is not perfectly equivalent because all numbers inawk are floating-point—in floating-point, ‘foo + 1 - 1’ doesnot necessarily equalfoo
. But the difference is minute aslong as you stick to numbers that are fairly small (less than 10e12).
Fields and array elements are incrementedjust like variables. (Use ‘$(i++)’ when you want to do a field referenceand a variable increment at the same time. The parentheses are necessarybecause of the precedence of the field reference operator ‘$’.)
The decrement operator ‘--’ works just like ‘++’, except thatit subtracts one instead of adding it. As with ‘++’, it can be used beforethe lvalue to pre-decrement or after it to post-decrement. Following is a summary of increment and decrement expressions:
++
lvalue
++
--
lvalue
--
Doctor, doctor! It hurts when I do this!
So don't do that!
Groucho Marx
What happens for something like the following?
b = 6 print b += b++
Or something even stranger?
b = 6 b += ++b + b++ print b
In other words, when do the various side effects prescribed by thepostfix operators (‘b++’) take effect? When side effects happen isimplementation defined. In other words, it is up to the particular version ofawk. The result for the first example may be 12 or 13, and for the second, itmay be 22 or 23.
In short, doing things like this is not recommended and definitelynot anything that you can rely upon for portability. You should avoid such things in your own programs.
In certain contexts, expression values also serve as “truth values;” i.e.,they determine what should happen next as the program runs. Thissection describes howawk defines “true” and “false”and how values are compared.
Many programming languages have a special representation for the conceptsof “true” and “false.” Such languages usually use the specialconstantstrue
and false
, or perhaps their uppercaseequivalents. However,awk is different. It borrows a very simple concept of true andfalse from C. Inawk, any nonzero numeric value or anynonempty string value is true. Any other value (zero or the nullstring,""
) is false. The following program prints ‘A strangetruth value’ three times:
BEGIN { if (3.1415927) print "A strange truth value" if ("Four Score And Seven Years Ago") print "A strange truth value" if (j = 57) print "A strange truth value" }
There is a surprising consequence of the “nonzero or non-null” rule:the string constant"0"
is actually true, because it is non-null. (d.c.)
The Guide is definitive. Reality is frequently inaccurate.
The Hitchhiker's Guide to the Galaxy
Unlike other programming languages, awk variables do not have afixed type. Instead, they can be either a number or a string, dependingupon the value that is assigned to them. We look now at how variables are typed, and howawkcompares variables.
The 1992 POSIX standard introducedthe concept of anumeric string, which is simply a string that lookslike a number—for example," +2"
. This concept is usedfor determining the type of a variable. The type of the variable is important because the types of two variablesdetermine how they are compared. The various versions of the POSIX standard did not get the rulesquite right for several editions. Fortunately, as of at least the2008 standard (and possibly earlier), the standard has been fixed,and variable typing follows these rules:31
getline
input, FILENAME
, ARGV
elements,ENVIRON
elements, and the elements of an array created bypatsplit()
,split()
and match()
that are numericstrings have the strnum attribute. Otherwise, they havethe string attribute. Uninitialized variables also have thestrnum attribute.The last rule is particularly important. In the following program,a
has numeric type, even though it is later used in a stringoperation:
BEGIN { a = 12.345 b = a " is a cute number" print b }
When two operands are compared, either string comparison or numeric comparisonmay be used. This depends upon the attributes of the operands, according to thefollowing symmetric matrix:
+——————————————————————– | STRING NUMERIC STRNUM ———–+——————————————————————– | STRING | string string string | NUMERIC | string numeric numeric | STRNUM | string numeric numeric ———–+——————————————————————–
The basic idea is that user input that looks numeric—and onlyuser input—should be treated as numeric, even though it is actuallymade of characters and is therefore also a string. Thus, for example, the string constant" +3.14"
,when it appears in program source code,is a string—even though it looks numeric—andisnever treated as number for comparisonpurposes.
In short, when one operand is a “pure” string, such as a stringconstant, then a string comparison is performed. Otherwise, anumeric comparison is performed.
This point bears additional emphasis: All user input is made of characters,and so is first and foremost ofstring type; input stringsthat look numeric are additionally given thestrnum attribute. Thus, the six-character input string ‘ +3.14’ receives thestrnum attribute. In contrast, the eight-character literal" +3.14"
appearing in program text is a string constant. The following examples print ‘1’ when the comparison betweenthe two different constants is true, ‘0’ otherwise:
$ echo ' +3.14' | gawk '{ print $0 == " +3.14" }' True -| 1 $ echo ' +3.14' | gawk '{ print $0 == "+3.14" }' False -| 0 $ echo ' +3.14' | gawk '{ print $0 == "3.14" }' False -| 0 $ echo ' +3.14' | gawk '{ print $0 == 3.14 }' True -| 1 $ echo ' +3.14' | gawk '{ print $1 == " +3.14" }' False -| 0 $ echo ' +3.14' | gawk '{ print $1 == "+3.14" }' True -| 1 $ echo ' +3.14' | gawk '{ print $1 == "3.14" }' False -| 0 $ echo ' +3.14' | gawk '{ print $1 == 3.14 }' True -| 1
Comparison expressions compare strings or numbers forrelationships such as equality. They are written usingrelationaloperators, which are a superset of those in C. table-relational-ops describes them.
Expression | Result |
---|---|
x < y |
True if x is less than y. |
x <= y |
True if x is less than or equal to y. |
x > y |
True if x is greater than y. |
x >= y |
True if x is greater than or equal to y. |
x == y |
True if x is equal to y. |
x != y |
True if x is not equal to y. |
x ~ y |
True if the string x matches the regexp denoted byy. |
x !~ y |
True if the string x does not match the regexp denoted byy. |
subscript in array |
True if the array array has an element with the subscriptsubscript. |
Table 6.3: Relational Operators
Comparison expressions have the value one if true and zero if false. When comparing operands of mixed types, numeric operands are convertedto strings using the value ofCONVFMT
(see Conversion).
Strings are comparedby comparing the first character of each, then the second character of each,and so on. Thus,"10"
is less than "9"
. If there are twostrings where one is a prefix of the other, the shorter string is less thanthe longer one. Thus,"abc"
is less than "abcd"
.
It is very easy to accidentally mistype the ‘==’ operator andleave off one of the ‘=’ characters. The result is still validawk code, but the program does not do what is intended:
if (a = b) # oops! should be a == b ... else ...
Unless b
happens to be zero or the null string, theif
part of the test always succeeds. Because the operators areso similar, this kind of error is very difficult to spot whenscanning the source code.
The following table of expressions illustrates the kind of comparisongawk performs, as well as what the result of the comparison is:
1.5 <= 2.0
"abc" >= "xyz"
1.5 != " +2"
"1e2" < "3"
a = 2; b = "2"
a == b
a = 2; b = " +2"
a == b
In this example:
$ echo 1e2 3 | awk '{ print ($1 < $2) ? "true" : "false" }' -| false
the result is ‘false’ because both$1
and $2
are user input. They are numeric strings—therefore both havethestrnum attribute, dictating a numeric comparison. The purpose of the comparison rules and the use of numeric strings isto attempt to produce the behavior that is “least surprising,” whilestill “doing the right thing.”
String comparisons and regular expression comparisons are very different. For example:
x == "foo"
has the value one, or is true if the variable x
is precisely ‘foo’. By contrast:
x ~ /foo/
has the value one if x
contains ‘foo’, such as"Oh, what a fool am I!"
.
The righthand operand of the ‘~’ and ‘!~’ operators may beeither a regexp constant (/.../
) or an ordinaryexpression. In the latter case, the value of the expression as a string is used as adynamic regexp (see Regexp Usage; alsoseeComputed Regexps).
In modern implementations ofawk, a constant regularexpression in slashes by itself is also an expression. The regexp/
regexp/
is an abbreviation for the following comparison expression:
$0 ~ /regexp/
One special place where /foo/
is not an abbreviation for‘$0 ~ /foo/’ is when it is the righthand operand of ‘~’ or‘!~’. See Using Constant Regexps,where this is discussed in more detail.
The POSIX standard says that string comparison is performed basedon the locale's collating order. This is usually very differentfrom the results obtained when doing straight character-by-charactercomparison.32
Because this behavior differs considerably from existing practice,gawk only implements it when in POSIX mode (seeOptions). Here is an example to illustrate the difference, in an ‘en_US.UTF-8’locale:
$ gawk 'BEGIN { printf("ABC < abc = %s\n", > ("ABC" < "abc" ? "TRUE" : "FALSE")) }' -| ABC < abc = TRUE $ gawk --posix 'BEGIN { printf("ABC < abc = %s\n", > ("ABC" < "abc" ? "TRUE" : "FALSE")) }' -| ABC < abc = FALSE
ABoolean expression is a combination of comparison expressions ormatching expressions, using the Boolean operators “or”(‘||’), “and” (‘&&’), and “not” (‘!’), along withparentheses to control nesting. The truth value of the Boolean expression iscomputed by combining the truth values of the component expressions. Boolean expressions are also referred to aslogical expressions. The terms are equivalent.
Boolean expressions can be used wherever comparison and matchingexpressions can be used. They can be used inif
, while
,do
, and for
statements(seeStatements). They have numeric values (one if true, zero if false) that come into playif the result of the Boolean expression is stored in a variable orused in arithmetic.
In addition, every Boolean expression is also a valid pattern, soyou can use one as a pattern to control the execution of rules. The Boolean operators are:
&&
boolean2
if ($0 ~ /2400/ && $0 ~ /foo/) print
The subexpression boolean2 is evaluated only if boolean1is true. This can make a difference whenboolean2 containsexpressions that have side effects. In the case of ‘$0 ~ /foo/ &&($2 == bar++)’, the variablebar
is not incremented if there isno substring ‘foo’ in the record.
||
boolean2
if ($0 ~ /2400/ || $0 ~ /foo/) print
The subexpression boolean2 is evaluated only if boolean1is false. This can make a difference whenboolean2 containsexpressions that have side effects.
!
boolean
BEGIN { if (! ("HOME" in ENVIRON)) print "no home!" }
(The in
operator is described inReference to Elements.)
The ‘&&’ and ‘||’ operators are calledshort-circuitoperators because of the way they work. Evaluation of the full expressionis “short-circuited” if the result can be determined part way throughits evaluation.
Statements that use ‘&&’ or ‘||’ can be continued simplyby putting a newline after them. But you cannot put a newline in frontof either of these operators without using backslash continuation(see Statements/Lines).
The actual value of an expression using the ‘!’ operator iseither one or zero, depending upon the truth value of the expression itis applied to. The ‘!’ operator is often useful for changing the sense of a flagvariable from false to true and back again. For example, the followingprogram is one way to print lines in between special bracketing lines:
$1 == "START" { interested = ! interested; next } interested == 1 { print } $1 == "END" { interested = ! interested; next }
The variable interested
, as with all awk variables, startsout initialized to zero, which is also false. When a line is seen whosefirst field is ‘START’, the value of interested
is toggledto true, using ‘!’. The next rule prints lines as long asinterested
is true. When a line is seen whose first field is‘END’,interested
is toggled back to false.33
NOTE: Thenext
statement is discussed in Next Statement.next
tells awk to skip the rest of the rules, get thenext record, and start processing the rules over again at the top. The reason it's there is to avoid printing the bracketing‘ START’ and ‘ END’ lines.
Aconditional expression is a special kind of expression that hasthree operands. It allows you to use one expression's value to selectone of two other expressions. The conditional expression is the same as in the C language,as shown here:
selector ? if-true-exp : if-false-exp
There are three subexpressions. The first, selector, is alwayscomputed first. If it is “true” (not zero or not null), thenif-true-exp is computed next and its value becomes the value ofthe whole expression. Otherwise,if-false-exp is computed nextand its value becomes the value of the whole expression. For example, the following expression produces the absolute value ofx
:
x >= 0 ? x : -x
Each time the conditional expression is computed, only one ofif-true-exp andif-false-exp is used; the other is ignored. This is important when the expressions have side effects. For example,this conditional expression examines elementi
of either arraya
or array b
, and incrementsi
:
x == y ? a[i++] : b[i++]
This is guaranteed to increment i
exactly once, because each timeonly one of the two increment expressions is executedand the other is not. SeeArrays,for more information about arrays.
As a minor gawk extension,a statement that uses ‘?:’ can be continued simplyby putting a newline after either character. However, putting a newline in frontof either character does not work without using backslash continuation(see Statements/Lines). If --posix is specified(seeOptions), then this extension is disabled.
A function is a name for a particular calculation. This enables you toask for it by name at any point in the program. Forexample, the functionsqrt()
computes the square root of a number.
A fixed set of functions arebuilt-in, which means they areavailable in every awk program. Thesqrt()
function is oneof these. See Built-in, for a list of built-infunctions and their descriptions. In addition, you can definefunctions for use in your program. SeeUser-defined,for instructions on how to do this.
The way to use a function is with afunction call expression,which consists of the function name followed immediately by a list ofarguments in parentheses. The arguments are expressions thatprovide the raw materials for the function's calculations. When there is more than one argument, they are separated by commas. Ifthere are no arguments, just write ‘()’ after the function name. The following examples show function calls with and without arguments:
sqrt(x^2 + y^2) one argument atan2(y, x) two arguments rand() no arguments
CAUTION: Do not put any space between the function name and the open-parenthesis! A user-defined function name looks just like the name of avariable—a space would make the expression look like concatenation ofa variable with an expression inside parentheses. With built-in functions, space before the parenthesis is harmless, butit is best not to get into the habit of using space to avoid mistakeswith user-defined functions.
Each function expects a particular numberof arguments. For example, the sqrt()
function must be called witha single argument, the number of which to take the square root:
sqrt(argument)
Some of the built-in functions have one ormore optional arguments. If those arguments are not supplied, the functionsuse a reasonable default value. SeeBuilt-in, for full details. If argumentsare omitted in calls to user-defined functions, then those arguments aretreated as local variables and initialized to the empty string(seeUser-defined).
As an advanced feature, gawk provides indirect function calls,which is a way to choose the function to call at runtime, instead ofwhen you write the source code to your program. We defer discussion ofthis feature until later; see Indirect Calls.
Like every other expression, the function call has a value, which iscomputed by the function based on the arguments you give it. In thisexample, the value of ‘sqrt(argument)’ is the square root ofargument. The following program reads numbers, one number per line, and prints thesquare root of each one:
$ awk '{ print "The square root of", $1, "is", sqrt($1) }' 1 -| The square root of 1 is 1 3 -| The square root of 3 is 1.73205 5 -| The square root of 5 is 2.23607 Ctrl-d
A function can also have side effects, such as assigningvalues to certain variables or doing I/O. This program shows how thematch()
function(see String Functions)changes the variablesRSTART
and RLENGTH
:
{ if (match($1, $2)) print RSTART, RLENGTH else print "no match" }
Here is a sample run:
$ awk -f matchit.awk aaccdd c+ -| 3 2 foo bar -| no match abcdefg e -| 5 1
Operator precedence determines how operators are grouped whendifferent operators appear close by in one expression. For example,‘*’ has higher precedence than ‘+’; thus, ‘a + b * c’means to multiplyb
and c
, and then add a
to theproduct (i.e., ‘a + (b * c)’).
The normal precedence of the operators can be overruled by using parentheses. Think of the precedence rules as saying where theparentheses are assumed to be. Infact, it is wise to always use parentheses whenever there is an unusualcombination of operators, because other people who read the program maynot remember what the precedence is in this case. Even experienced programmers occasionally forget the exact rules,which leads to mistakes. Explicit parentheses help preventany such mistakes.
When operators of equal precedence are used together, the leftmostoperator groups first, except for the assignment, conditional, andexponentiation operators, which group in the opposite order. Thus, ‘a - b + c’ groups as ‘(a - b) + c’ and‘a = b = c’ groups as ‘a = (b = c)’.
Normally the precedence of prefix unary operators does not matter,because there is only one way to interpretthem: innermost first. Thus, ‘$++i’ means ‘$(++i)’ and‘++$x’ means ‘++($x)’. However, when another operator followsthe operand, then the precedence of the unary operators can matter. ‘$x^2’ means ‘($x)^2’, but ‘-x^2’ means‘-(x^2)’, because ‘-’ has lower precedence than ‘^’,whereas ‘$’ has higher precedence. Also, operators cannot be combined in a way that violates theprecedence rules; for example, ‘$$0++--’ is not a validexpression because the first ‘$’ has higher precedence than the‘++’; to avoid the problem the expression can be rewritten as‘$($0++)--’.
This table presents awk's operators, in order of highestto lowest precedence:
(...)
$
++ --
^ **
+ - !
* / %
+ -
< <= == != > >= >> | |&
Note that the I/O redirection operators inprint
and printf
statements belong to the statement level, not to expressions. Theredirection does not produce an expression that could be the operand ofanother operator. As a result, it does not make sense to use aredirection operator near another operator of lower precedence withoutparentheses. Such combinations (for example, ‘print foo > a ? b : c’),result in syntax errors. The correct way to write this statement is ‘print foo > (a ? b : c)’.
~ !~
in
&&
||
?:
= += -= *= /= %= ^= **=
NOTE: The ‘ |&’, ‘ **’, and ‘ **=’ operators are not specified by POSIX. For maximum portability, do not use them.
Modern systems support the notion oflocales: a way to tellthe system about the local character set and language.
Once upon a time, the locale setting used to affect regexp matching(see Ranges and Locales), but this is no longer true.
Locales can affect record splitting. For the normal case of ‘RS = "\n"’, the locale is largely irrelevant. For other single-character record separators, setting ‘LC_ALL=C’in the environmentwill give you much better performance when reading records. Otherwise,gawk has to make several function calls,per inputcharacter, to find the record terminator.
According to POSIX, string comparison is also affected by locales(similar to regular expressions). The details are presented inPOSIX String Comparison.
Finally, the locale affects the value of the decimal point characterused when gawk parses input data. This is discussed indetail inConversion.
As you have already seen, each awk statement consists ofa pattern with an associated action. This chapter describes howyou build patterns and actions, what kinds of things you can do withinactions, and awk's built-in variables.
The pattern-action rules and the statements available for usewithin actions form the core ofawk programming. In a sense, everything coveredup to here has been the foundationthat programs are built on top of. Now it's time to startbuilding something useful.
Patterns in awk control the execution of rules—a rule isexecuted when its pattern matches the current input record. The following is a summary of the types ofawk patterns:
/
regular expression
/
,
pat2
BEGIN
END
BEGINFILE
ENDFILE
Regular expressions are one of the first kinds of patterns presentedin this book. This kind of pattern is simply a regexp constant in the pattern part ofa rule. Its meaning is ‘$0 ~ /pattern/’. The pattern matches when the input record matches the regexp. For example:
/foo|bar|baz/ { buzzwords++ } END { print buzzwords, "buzzwords seen" }
Any awk expression is valid as anawk pattern. The pattern matches if the expression's value is nonzero (if anumber) or non-null (if a string). The expression is reevaluated each time the rule is tested against a newinput record. If the expression uses fields such as $1
, thevalue depends directly on the new input record's text; otherwise, itdepends on only what has happened so far in the execution of theawk program.
Comparison expressions, using the comparison operators described inTyping and Comparison,are a very common kind of pattern. Regexp matching and nonmatching are also very common expressions. The left operand of the ‘~’ and ‘!~’ operators is a string. The right operand is either a constant regular expression enclosed inslashes (/
regexp/
), or any expression whose string valueis used as a dynamic regular expression(seeComputed Regexps). The following example prints the second field of each input recordwhose first field is precisely ‘foo’:
$ awk '$1 == "foo" { print $2 }' BBS-list
(There is no output, because there is no BBS site with the exact name ‘foo’.) Contrast this with the following regular expression match, whichaccepts any record with a first field that contains ‘foo’:
$ awk '$1 ~ /foo/ { print $2 }' BBS-list -| 555-1234 -| 555-6699 -| 555-6480 -| 555-2127
A regexp constant as a pattern is also a special case of an expressionpattern. The expression/foo/
has the value one if ‘foo’appears in the current input record. Thus, as a pattern,/foo/
matches any record containing ‘foo’.
Boolean expressions are also commonly used as patterns. Whether the patternmatches an input record depends on whether its subexpressions match. For example, the following command prints all the records inBBS-list that contain both ‘2400’ and ‘foo’:
$ awk '/2400/ && /foo/' BBS-list -| fooey 555-1234 2400/1200/300 B
The following command prints all records inBBS-list that containeither ‘2400’ or ‘foo’(or both, of course):
$ awk '/2400/ || /foo/' BBS-list -| alpo-net 555-3412 2400/1200/300 A -| bites 555-1675 2400/1200/300 A -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sdace 555-3430 2400/1200/300 A -| sabafoo 555-2127 1200/300 C
The following command prints all records inBBS-list that donot contain the string ‘foo’:
$ awk '! /foo/' BBS-list -| aardvark 555-5553 1200/300 B -| alpo-net 555-3412 2400/1200/300 A -| barfly 555-7685 1200/300 A -| bites 555-1675 2400/1200/300 A -| camelot 555-0542 300 C -| core 555-2912 1200/300 C -| sdace 555-3430 2400/1200/300 A
The subexpressions of a Boolean operator in a pattern can be constant regularexpressions, comparisons, or any otherawk expressions. Rangepatterns are not expressions, so they cannot appear inside Booleanpatterns. Likewise, the special patternsBEGIN
, END
,BEGINFILE
and ENDFILE
,which never match any input record, are not expressions and cannotappear inside Boolean patterns.
Arange pattern is made of two patterns separated by a comma, inthe form ‘begpat,endpat’. It is used to match ranges ofconsecutive input records. The first pattern,begpat, controlswhere the range begins, while endpat controls wherethe pattern ends. For example, the following:
awk '$1 == "on", $1 == "off"' myfile
prints every record in myfile between ‘on’/‘off’ pairs, inclusive.
A range pattern starts out by matching begpat against everyinput record. When a record matchesbegpat, the range pattern isturned on and the range pattern matches this record as well. As long asthe range pattern stays turned on, it automatically matches every inputrecord read. The range pattern also matchesendpat against everyinput record; when this succeeds, the range pattern is turned off againfor the following record. Then the range pattern goes back to checkingbegpat against each record.
The record that turns on the range pattern and the one that turns itoff both match the range pattern. If you don't want to operate onthese records, you can write if
statements in the rule's actionto distinguish them from the records you are interested in.
It is possible for a pattern to be turned on and off by the samerecord. If the record satisfies both conditions, then the action isexecuted for just that record. For example, suppose there is text between two identical markers (e.g.,the ‘%’ symbol), each on its own line, that should be ignored. A first attempt would be tocombine a range pattern that describes the delimited text with thenext
statement(not discussed yet, seeNext Statement). This causes awk to skip any further processing of the currentrecord and start over again with the next input record. Such a programlooks like this:
/^%$/,/^%$/ { next } { print }
This program fails because the range pattern is both turned on and turned offby the first line, which just has a ‘%’ on it. To accomplish this task,write the program in the following manner, using a flag:
/^%$/ { skip = ! skip; next } skip == 1 { next } # skip lines with `skip' set
In a range pattern, the comma (‘,’) has the lowest precedence ofall the operators (i.e., it is evaluated last). Thus, the followingprogram attempts to combine a range pattern with another, simpler test:
echo Yes | awk '/1/,/2/ || /Yes/'
The intent of this program is ‘(/1/,/2/) || /Yes/’. However,awk interprets this as ‘/1/, (/2/ || /Yes/)’. This cannot be changed or worked around; range patterns do not combinewith other patterns:
$ echo Yes | gawk '(/1/,/2/) || /Yes/' error--> gawk: cmd. line:1: (/1/,/2/) || /Yes/ error--> gawk: cmd. line:1: ^ syntax error
BEGIN
and END
Special PatternsAll the patterns described so far are for matching input records. TheBEGIN
and END
special patterns are different. They supply startup and cleanup actions forawk programs. BEGIN
and END
rules must have actions; there is no defaultaction for these rules because there is no current record when they run.BEGIN
and END
rules are often referred to as“BEGIN
andEND
blocks” by long-time awkprogrammers.
A BEGIN
rule is executed once only, before the first input recordis read. Likewise, anEND
rule is executed once only, after all theinput is read. For example:
$ awk ' > BEGIN { print "Analysis of \"foo\"" } > /foo/ { ++n } > END { print "\"foo\" appears", n, "times." }' BBS-list -| Analysis of "foo" -| "foo" appears 4 times.
This program finds the number of records in the input fileBBS-listthat contain the string ‘foo’. TheBEGIN
rule prints a titlefor the report. There is no need to use theBEGIN
rule toinitialize the counter n
to zero, since awk does thisautomatically (see Variables). The second rule increments the variable n
every time arecord containing the pattern ‘foo’ is read. TheEND
ruleprints the value of n
at the end of the run.
The special patterns BEGIN
and END
cannot be used in rangesor with Boolean operators (indeed, they cannot be used with any operators). Anawk program may have multiple BEGIN
and/orEND
rules. They are executed in the order in which they appear: all theBEGIN
rules at startup and all the END
rules at termination.BEGIN
and END
rules may be intermixed with other rules. This feature was added in the 1987 version ofawk and is includedin the POSIX standard. The original (1978) version ofawkrequired the BEGIN
rule to be placed at the beginning of theprogram, theEND
rule to be placed at the end, and only allowed one ofeach. This is no longer required, but it is a good idea to follow this templatein terms of program organization and readability.
Multiple BEGIN
and END
rules are useful for writinglibrary functions, because each library file can have its ownBEGIN
and/orEND
rule to do its own initialization and/or cleanup. The order in which library functions are named on the command linecontrols the order in which theirBEGIN
and END
rules areexecuted. Therefore, you have to be careful when writing such rules inlibrary files so that the order in which they are executed doesn't matter. SeeOptions, for more information onusing library functions. SeeLibrary Functions,for a number of useful library functions.
If an awk program has only BEGIN
rules and noother rules, then the program exits after theBEGIN
rule isrun.34 However, if anEND
rule exists, then the input is read, even if there areno other rules in the program. This is necessary in case theEND
rule checks the FNR
and NR
variables.
BEGIN
and END
RulesThere are several (sometimes subtle) points to remember when doing I/Ofrom aBEGIN
or END
rule. The first has to do with the value of$0
in a BEGIN
rule. Because BEGIN
rules are executed before any input is read,there simply is no input record, and therefore no fields, whenexecutingBEGIN
rules. References to $0
and the fieldsyield a null string or zero, depending upon the context. One wayto give$0
a real value is to execute a getline
commandwithout a variable (seeGetline). Another way is simply to assign a value to $0
.
The second point is similar to the first but from the other direction. Traditionally, due largely to implementation issues,$0
andNF
were undefined inside an END
rule. The POSIX standard specifies thatNF
is available in an END
rule. It contains the number of fields from the last input record. Most probably due to an oversight, the standard does not say that$0
is also preserved, although logically one would think that it should be. In fact,gawk does preserve the value of $0
for use inEND
rules. Be aware, however, that Brian Kernighan'sawk, and possiblyother implementations, do not.
The third point follows from the first two. The meaning of ‘print’inside aBEGIN
or END
rule is the same as always:‘print $0’. If$0
is the null string, then this prints anempty record. Many long timeawk programmers use an unadorned‘print’ inBEGIN
and END
rules, to mean ‘print ""’,relying on$0
being null. Although one might generally get away withthis in BEGIN
rules, it is a very bad idea in END
rules,at least in gawk. It is also poor style, since if an emptyline is needed in the output, the program should print one explicitly.
Finally, the next
and nextfile
statements are not allowedin a BEGIN
rule, because the implicitread-a-record-and-match-against-the-rules loop has not started yet. Similarly, those statementsare not valid in anEND
rule, since all the input has been read. (See Next Statement, and seeNextfile Statement.)
BEGINFILE
and ENDFILE
Special PatternsThis section describes agawk-specific feature.
Two special kinds of rule, BEGINFILE
and ENDFILE
, giveyou “hooks” intogawk's command-line file processing loop. As with theBEGIN
and END
rules (see BEGIN/END), allBEGINFILE
rules in a program are merged, in the order they areread bygawk, and all ENDFILE
rules are merged as well.
The body of the BEGINFILE
rules is executed just beforegawk reads the first record from a file.FILENAME
is set to the name of the current file, and FNR
is set to zero.
The BEGINFILE
rule provides you the opportunity for two tasksthat would otherwise be difficult or impossible to perform:
You do this by checking if the ERRNO
variable is not the emptystring; if so, thengawk was not able to open the file. Inthis case, your program can execute thenextfile
statement(see Nextfile Statement). This causesgawk to skipthe file entirely. Otherwise,gawk exits with the usualfatal error.
The ENDFILE
rule is called when gawk has finished processingthe last record in an input file. For the last input file,it will be called before anyEND
rules.
Normally, when an error occurs when reading input in the normal inputprocessing loop, the error is fatal. However, if anENDFILE
rule is present, the error becomes non-fatal, and instead ERRNO
is set. This makes it possible to catch and process I/O errors at thelevel of theawk program.
Thenext
statement (see Next Statement) is not allowed insideeither aBEGINFILE
or and ENDFILE
rule. The nextfile
statement (seeNextfile Statement) is allowed only inside aBEGINFILE
rule, but not inside anENDFILE
rule.
Thegetline
statement (see Getline) is restricted insidebothBEGINFILE
and ENDFILE
. Only the ‘getlinevariable <file’ form is allowed.
BEGINFILE
and ENDFILE
are gawk extensions. In most otherawk implementations, or if gawk is incompatibility mode (seeOptions), they are not special.
An empty (i.e., nonexistent) pattern is considered to matcheveryinput record. For example, the program:
awk '{ print $1 }' BBS-list
prints the first field of every record.
awk programs are often used as components in largerprograms written in shell. For example, it is very common to use a shell variable tohold a pattern that theawk program searches for. There are two ways to get the value of the shell variableinto the body of theawk program.
The most common method is to use shell quoting to substitutethe variable's value into the program inside the script. For example, in the following program:
printf "Enter search pattern: " read pattern awk "/$pattern/ "'{ nmatches++ } END { print nmatches, "found" }' /path/to/data
the awk program consists of two pieces of quoted textthat are concatenated together to form the program. The first part is double-quoted, which allows substitution ofthepattern
shell variable inside the quotes. The second part is single-quoted.
Variable substitution via quoting works, but can be potentiallymessy. It requires a good understanding of the shell's quoting rules(seeQuoting),and it's often difficult to correctlymatch up the quotes when reading the program.
A better method is to use awk's variable assignment feature(seeAssignment Options)to assign the shell variable's value to anawk variable'svalue. Then use dynamic regexps to match the pattern(seeComputed Regexps). The following shows how to redo theprevious example using this technique:
printf "Enter search pattern: " read pattern awk -v pat="$pattern" '$0 ~ pat { nmatches++ } END { print nmatches, "found" }' /path/to/data
Now, the awk program is just one single-quoted string. The assignment ‘-v pat="$pattern"’ still requires double quotes,in case there is whitespace in the value of $pattern
. The awk variablepat
could be named pattern
too, but that would be more confusing. Using a variable alsoprovides more flexibility, since the variable can be used anywhere insidethe program—for printing, as an array subscript, or for any otheruse—without requiring the quoting tricks at every point in the program.
An awk program or script consists of a series ofrules and function definitions interspersed. (Functions aredescribed later. SeeUser-defined.) A rule contains a pattern and an action, either of which (but notboth) may be omitted. The purpose of theaction is to tellawk what to do once a match for the pattern is found. Thus,in outline, anawk program generally looks like this:
[pattern] { action } pattern [{ action }] ... function name(args) { ... } ...
An action consists of one or more awk statements, enclosedin curly braces (‘{...}’). Each statement specifies onething to do. The statements are separated by newlines or semicolons. The curly braces around an action must be used even if the actioncontains only one statement, or if it contains no statements atall. However, if you omit the action entirely, omit the curly braces aswell. An omitted action is equivalent to ‘{ print $0 }’:
/foo/ { } matchfoo
, do nothing --- empty action /foo/ matchfoo
, print the record --- omitted action
The following types of statements are supported in awk:
if
,
for
,
while
, and
do
) as well as a fewspecial ones (see Statements).
if
,
while
,
do
,or
for
statement.
getline
command(see Getline). Also supplied in
awk are the
next
statement (see Next Statement),and the
nextfile
statement(see Nextfile Statement).
print
and
printf
. See Printing.
Control statements, such asif
, while
, and so on,control the flow of execution in awk programs. Most of awk'scontrol statements are patterned after similar statements in C.
All the control statements start with special keywords, such as if
and while
, to distinguish them from simple expressions. Many control statements contain other statements. For example, theif
statement contains another statement that may or may not beexecuted. The contained statement is called thebody. To include more than one statement in the body, group them into asinglecompound statement with curly braces, separating them withnewlines or semicolons.
if
-else
StatementThe if
-else
statement isawk's decision-makingstatement. It looks like this:
if (condition) then-body [else else-body]
The condition is an expression that controls what the rest of thestatement does. If thecondition is true, then-body isexecuted; otherwise, else-body is executed. Theelse
part of the statement isoptional. The condition is considered false if its value is zero orthe null string; otherwise, the condition is true. Refer to the following:
if (x % 2 == 0) print "x is even" else print "x is odd"
In this example, if the expression ‘x % 2 == 0’ is true (that is,if the value ofx
is evenly divisible by two), then the firstprint
statement is executed; otherwise, the secondprint
statement is executed. If the else
keyword appears on the same line asthen-body andthen-body is not a compound statement (i.e., not surrounded bycurly braces), then a semicolon must separatethen-body fromthe else
. To illustrate this, the previous example can be rewritten as:
if (x % 2 == 0) print "x is even"; else print "x is odd"
If the ‘;’ is left out,awk can't interpret the statement andit produces a syntax error. Don't actually write programs this way,because a human reader might fail to see theelse
if it is notthe first thing on its line.
while
StatementIn programming, aloop is a part of a program that canbe executed two or more times in succession. Thewhile
statement is the simplest looping statement inawk. It repeatedly executes a statement as long as a condition istrue. For example:
while (condition) body
body is a statement called thebody of the loop,and condition is an expression that controls how long the loopkeeps running. The first thing thewhile
statement does is test the condition. If the condition is true, it executes the statementbody. After body has been executed,condition is tested again, and if it is still true,body isexecuted again. This process repeats until the condition is no longertrue. If thecondition is initially false, the body of the loop isnever executed andawk continues with the statement followingthe loop. This example prints the first three fields of each record, one per line:
awk '{ i = 1 while (i <= 3) { print $i i++ } }' inventory-shipped
The body of this loop is a compound statement enclosed in braces,containing two statements. The loop works in the following manner: first, the value ofi
is set to one. Then, the while
statement tests whetheri
is less than or equal tothree. This is true when i
equals one, so thei
-thfield is printed. Then the ‘i++’ increments the value ofi
and the loop repeats. The loop terminates when i
reaches four.
A newline is not required between the condition and thebody; however using one makes the program clearer unless the body is acompound statement or else is very simple. The newline after the open-bracethat begins the compound statement is not required either, but theprogram is harder to read without it.
do
-while
StatementThedo
loop is a variation of the while
looping statement. Thedo
loop executes the body once and then repeats thebody as long as thecondition is true. It looks like this:
do body while (condition)
Even if the condition is false at the start, the body isexecuted at least once (and only once, unless executingbodymakes condition true). Contrast this with the correspondingwhile
statement:
while (condition) body
This statement does not execute body even once if theconditionis false to begin with. The following is an example of a do
statement:
{ i = 1 do { print $0 i++ } while (i <= 10) }
This program prints each input record 10 times. However, it isn't a veryrealistic example, since in this case an ordinarywhile
would dojust as well. This situation reflects actual experience; onlyoccasionally is there a real use for ado
statement.
for
StatementThe for
statement makes it more convenient to count iterations of aloop. The general form of thefor
statement looks like this:
for (initialization; condition; increment) body
The initialization, condition, and increment parts arearbitrary awk expressions, andbody stands for anyawk statement.
The for
statement starts by executing initialization. Then, as longas thecondition is true, it repeatedly executes body and thenincrement. Typically,initialization sets a variable toeither zero or one, increment adds one to it, andconditioncompares it against the desired number of iterations. For example:
awk '{ for (i = 1; i <= 3; i++) print $i }' inventory-shipped
This prints the first three fields of each input record, with one field perline.
It isn't possible toset more than one variable in theinitialization part without using a multiple assignment statementsuch as ‘x = y = 0’. This makes sense only if all the initial valuesare equal. (But it is possible to initialize additional variables by writingtheir assignments as separate statements preceding thefor
loop.)
The same is true of the increment part. Incrementing additionalvariables requires separate statements at the end of the loop. The C compound expression, using C's comma operator, is useful inthis context but it is not supported inawk.
Most often, increment is an increment expression, as in the previousexample. But this is not required; it can be any expressionwhatsoever. For example, the following statement prints all the powers of twobetween 1 and 100:
for (i = 1; i <= 100; i *= 2) print i
If there is nothing to be done, any of the three expressions in theparentheses following thefor
keyword may be omitted. Thus,‘for (; x > 0;)’ is equivalent to ‘while (x > 0)’. If thecondition is omitted, it is treated as true, effectivelyyielding an infinite loop (i.e., a loop that never terminates).
In most cases, a for
loop is an abbreviation for a while
loop, as shown here:
initialization while (condition) { body increment }
The only exception is when thecontinue
statement(see Continue Statement) is usedinside the loop. Changing afor
statement to a while
statement in this way can change the effect of thecontinue
statement inside the loop.
The awk language has a for
statement in addition to awhile
statement because afor
loop is often both less work totype and more natural to think of. Counting the number of iterations isvery common in loops. It can be easier to think of this counting as partof looping rather than as something to do inside the loop.
There is an alternate version of thefor
loop, for iterating overall the indices of an array:
for (i in array) do something with array[i]
See Scanning an Array,for more information on this version of thefor
loop.
switch
StatementTheswitch
statement allows the evaluation of an expression andthe execution of statements based on acase
match. Case statementsare checked for a match in the order they are defined. If no suitablecase
is found, thedefault
section is executed, if supplied.
Each case
contains a single constant, be it numeric, string, orregexp. Theswitch
expression is evaluated, and then eachcase
's constant is compared against the result in turn. The type of constantdetermines the comparison: numeric or string do the usual comparisons. A regexp constant does a regular expression match against the stringvalue of the original expression. The general form of theswitch
statement looks like this:
switch (expression) { case value or regular expression: case-body default: default-body }
Control flow inthe switch
statement works as it does in C. Once a match to a givencase is made, the case statement bodies execute until abreak
,continue
, next
, nextfile
orexit
is encountered,or the end of the switch
statement itself. For example:
switch (NR * 2 + 1) { case 3: case "11": print NR - 1 break case /2[[:digit:]]+/: print NR default: print NR + 1 case -1: print NR * -1 }
Note that if none of the statements specified above halt executionof a matchedcase
statement, execution falls through to thenext case
until execution halts. In the above example, forany case value starting with ‘2’ followed by one or more digits,theprint
statement is executed and then falls through into thedefault
section, executing itsprint
statement. In turn,the −1 case will also be executed since thedefault
doesnot halt execution.
This switch
statement is a gawk extension. Ifgawk is in compatibility mode(see Options),it is not available.
break
StatementThebreak
statement jumps out of the innermost for
,while
, ordo
loop that encloses it. The following examplefinds the smallest divisor of any integer, and also identifies primenumbers:
# find smallest divisor of num { num = $1 for (div = 2; div * div <= num; div++) { if (num % div == 0) break } if (num % div == 0) printf "Smallest divisor of %d is %d\n", num, div else printf "%d is prime\n", num }
When the remainder is zero in the first if
statement, awkimmediatelybreaks out of the containing for
loop. This meansthat awk proceeds immediately to the statement following the loopand continues processing. (This is very different from theexit
statement, which stops the entire awk program. SeeExit Statement.)
The following program illustrates how the condition of a for
orwhile
statement could be replaced with a break
insideanif
:
# find smallest divisor of num { num = $1 for (div = 2; ; div++) { if (num % div == 0) { printf "Smallest divisor of %d is %d\n", num, div break } if (div * div > num) { printf "%d is prime\n", num break } } }
The break
statement is also used to break out of theswitch
statement. This is discussed inSwitch Statement.
Thebreak
statement has no meaning whenused outside the body of a loop orswitch
. However, although it was never documented,historical implementations ofawk treated the break
statement outside of a loop as if it were anext
statement(see Next Statement). (d.c.) Recent versions of Brian Kernighan'sawk no longer allow this usage,nor doesgawk.
continue
StatementSimilar to break
, the continue
statement is used only insidefor
,while
, and do
loops. It skipsover the rest of the loop body, causing the next cycle around the loopto begin immediately. Contrast this withbreak
, which jumps outof the loop altogether.
The continue
statement in a for
loop directs awk toskip the rest of the body of the loop and resume execution with theincrement-expression of thefor
statement. The following programillustrates this fact:
BEGIN { for (x = 0; x <= 20; x++) { if (x == 5) continue printf "%d ", x } print "" }
This program prints all the numbers from 0 to 20—except for 5, forwhich theprintf
is skipped. Because the increment ‘x++’is not skipped,x
does not remain stuck at 5. Contrast thefor
loop from the previous example with the followingwhile
loop:
BEGIN { x = 0 while (x <= 20) { if (x == 5) continue printf "%d ", x x++ } print "" }
This program loops forever once x
reaches 5.
Thecontinue
statement has no special meaning with respect to theswitch
statement, nor does it any meaning when used outside the body ofa loop. Historical versions ofawk treated a continue
statement outside a loop the same way they treated abreak
statement outside a loop: as if it were a next
statement(seeNext Statement). (d.c.) Recent versions of Brian Kernighan'sawk no longer work this way, nordoes gawk.
next
StatementThe next
statement forcesawk to immediately stop processingthe current record and go on to the next record. This means that nofurther rules are executed for the current record, and the rest of thecurrent rule's action isn't executed.
Contrast this with the effect of the getline
function(see Getline). That also causesawk to read the next record immediately, but it does not alter theflow of control in any way (i.e., the rest of the current action executeswith a new input record).
At the highest level,awk program execution is a loop that readsan input record and then tests each rule's pattern against it. If youthink of this loop as afor
statement whose body contains therules, then the next
statement is analogous to acontinue
statement. It skips to the end of the body of this implicit loop andexecutes the increment (which reads another record).
For example, suppose an awk program works only on recordswith four fields, and it shouldn't fail when given bad input. To avoidcomplicating the rest of the program, write a “weed out” rule nearthe beginning, in the following manner:
NF != 4 { err = sprintf("%s:%d: skipped: NF != 4\n", FILENAME, FNR) print err > "/dev/stderr" next }
Because of the next
statement,the program's subsequent rules won't see the bad record. The errormessage is redirected to the standard error output stream, as errormessages should be. For more detail seeSpecial Files.
If the next
statement causes the end of the input to be reached,then the code in anyEND
rules is executed. See BEGIN/END.
The next
statement is not allowed inside BEGINFILE
andENDFILE
rules. SeeBEGINFILE/ENDFILE.
According to the POSIX standard, the behavior is undefined ifthe next
statement is used in aBEGIN
or END
rule. gawk treats it as a syntax error. Although POSIX permits it,some otherawk implementations don't allow the next
statement inside function bodies(see User-defined). Just as with any othernext
statement, a next
statement inside afunction body reads the next record and starts processing it with thefirst rule in the program.
nextfile
Statementgawk provides the nextfile
statement,which is similar to the next
statement. (c.e.) However, instead of abandoning processing of the current record, thenextfile
statement instructsgawk to stop processing thecurrent data file.
The nextfile
statement is a gawk extension. In most otherawk implementations,or if gawk is in compatibility mode(seeOptions),nextfile
is not special.
Upon execution of the nextfile
statement,any ENDFILE
rules are executed,FILENAME
isupdated to the name of the next data file listed on the command line,FNR
is reset to one,ARGIND
is incremented,any BEGINFILE
rules are executed, and processingstarts over with the first rule in the program. (ARGIND
hasn't been introduced yet. SeeBuilt-in Variables.) If the nextfile
statement causes the end of the input to be reached,then the code in anyEND
rules is executed. See BEGIN/END.
The nextfile
statement is useful when there are many data filesto process but it isn't necessary to process every record in every file. Normally, in order to move on to the next data file, a programhas to continue scanning the unwanted records. The nextfile
statement accomplishes this much more efficiently.
In addition, nextfile
is useful inside a BEGINFILE
rule to skip over a file that would otherwise causegawkto exit with a fatal error. See BEGINFILE/ENDFILE.
While one might think that ‘close(FILENAME)’ would accomplishthe same asnextfile
, this isn't true. close()
isreserved for closing files, pipes, and coprocesses that areopened with redirections. It is not related to the main processing thatawk does with the files listed in ARGV
.
The current version of the Brian Kernighan's awk (see Other Versions) also supports nextfile
. However, it doesn't allow thenextfile
statement inside function bodies (seeUser-defined). gawk does; anextfile
inside a function body reads thenext record and starts processing it with the first rule in the program,just as any othernextfile
statement.
exit
StatementThe exit
statement causesawk to immediately stopexecuting the current rule and to stop processing input; any remaining inputis ignored. Theexit
statement is written as follows:
exit [return code]
When anexit
statement is executed from a BEGIN
rule, theprogram stops processing everything immediately. No input records areread. However, if anEND
rule is present,as part of executing the exit
statement,theEND
rule is executed(see BEGIN/END). Ifexit
is used in the body of an END
rule, it causesthe program to stop immediately.
An exit
statement that is not part of a BEGIN
or END
rule stops the execution of any further automatic rules for the currentrecord, skips reading any remaining input records, and executes theEND
rule if there is one. AnyENDFILE
rules are also skipped; they are not executed.
In such a case,if you don't want the END
rule to do its job, set a variableto nonzero before theexit
statement and check that variable inthe END
rule. SeeAssert Function,for an example that does this.
If an argument is supplied toexit
, its value is used as the exitstatus code for the awk process. If no argument is supplied,exit
causesawk to return a “success” status. In the case where an argumentis supplied to a firstexit
statement, and then exit
iscalled a second time from anEND
rule with no argument,awk uses the previously supplied exit value. (d.c.) SeeExit Status, for more information.
For example, suppose an error condition occurs that is difficult orimpossible to handle. Conventionally, programs report this byexiting with a nonzero status. Anawk program can do thisusing an exit
statement with a nonzero argument, as shownin the following example:
BEGIN { if (("date" | getline date_now) <= 0) { print "Can't get system date" > "/dev/stderr" exit 1 } print "current date is", date_now close("date") }
NOTE: For full portability, exit values should be between zero and 126, inclusive. Negative values, and values of 127 or greater, may not produce consistentresults across different operating systems.
Mostawk variables are available to use for your ownpurposes; they never change unless your program assigns values tothem, and they never affect anything unless your program examines them. However, a few variables inawk have special built-in meanings. awk examines some of these automatically, so that they enable youto tellawk how to do certain things. Others are setautomatically byawk, so that they carry information from theinternal workings ofawk to your program.
This section documents all the built-in variables ofgawk, most of which are also documented in the chaptersdescribing their areas of activity.
The following is an alphabetical list of variables that you can change tocontrol howawk does certain things. The variables that arespecific togawk are marked with a pound sign (‘#’).
BINMODE #
"r"
or
"w"
specify that input files andoutput files, respectively, should use binary I/O. A string value of
"rw"
or
"wr"
indicates that allfiles should use binary I/O. Any other string value is treated the same as
"rw"
,but causes
gawkto generate a warning message.
BINMODE
is described in more detail in PC Using.
This variable is agawk extension. In other awk implementations(exceptmawk,see Other Versions),or ifgawk is in compatibility mode(see Options),it is not special.
CONVFMT
sprintf()
function(see String Functions). Its default value is
"%.6g"
.
CONVFMT
was introduced by the POSIX standard.
FIELDWIDTHS #
FIELDWIDTHS
overrides the use of
FS
and
FPAT
for field splitting. See Constant Size, for more information.
If gawk is in compatibility mode(seeOptions), then FIELDWIDTHS
has no special meaning, and field-splitting operations occur basedexclusively on the value ofFS
.
FPAT #
FPAT
overrides the use of
FS
and
FIELDWIDTHS
for field splitting. See Splitting By Content, for more information.
If gawk is in compatibility mode(seeOptions), then FPAT
has no special meaning, and field-splitting operations occur basedexclusively on the value ofFS
.
FS
""
), then eachcharacter in the record becomes a separate field. (This behavior is a
gawk extension. POSIX
awk does notspecify the behavior when
FS
is the null string. Nonetheless, some other versions of
awk also treat
""
specially.)
The default value is" "
, a string consisting of a singlespace. As a special exception, this value means that anysequence of spaces, TABs, and/or newlines is a single separator.35 It also causesspaces, TABs, and newlines at the beginning and end of a record to be ignored.
You can set the value of FS
on the command line using the-F option:
awk -F, 'program' input-files
Ifgawk is using FIELDWIDTHS
orFPAT
for field splitting,assigning a value to FS
causes gawk to return tothe normal, FS
-based field splitting. An easy way to do thisis to simply say ‘FS = FS’, perhaps with an explanatory comment.
IGNORECASE #
IGNORECASE
is nonzero or non-null, then all string comparisonsand all regular expression matching are case independent. Thus, regexpmatching with ‘
~’ and ‘
!~’, as well as the
gensub()
,
gsub()
,
index()
,
match()
,
patsplit()
,
split()
, and
sub()
functions, record termination with
RS
, and field splitting with
FS
and
FPAT
, all ignore case when doing their particular regexp operations. However, the value of
IGNORECASE
does
not affect array subscriptingand it does not affect field splitting when using a single-characterfield separator. See Case-sensitivity.
If gawk is in compatibility mode(seeOptions),then IGNORECASE
has no special meaning. Thus, stringand regexp operations are always case-sensitive.
LINT #
"fatal"
, lint warnings become fatal errors. With a value of
"invalid"
, only warnings about things that areactually invalid are issued. (This is not fully implemented yet.) Any other true value prints nonfatal warnings. Assigning a false value to
LINT
turns off the lint warnings.
This variable is a gawk extension. It is not specialin otherawk implementations. Unlike the other special variables,changingLINT
does affect the production of lint warnings,even if gawk is in compatibility mode. Much asthe--lint and --traditional options independentlycontrol different aspects ofgawk's behavior, the controlof lint warnings during program execution is independent of the flavorofawk being executed.
OFMT
print
statement. It works by being passedas the first argument to the
sprintf()
function(see String Functions). Its default value is
"%.6g"
. Earlier versions of
awkalso used
OFMT
to specify the format for converting numbers tostrings in general expressions; this is now done by
CONVFMT
.
OFS
print
statement. Itsdefault value is
" "
, a string consisting of a single space.
ORS
print
statement. Its default value is
"\n"
, the newlinecharacter. (See Output Separators.)
RS
The ability for RS
to be a regular expressionis a gawk extension. In most otherawk implementations,or if gawk is in compatibility mode(seeOptions),just the first character of RS
's value is used.
SUBSEP
"\034"
and is used to separate the parts of the indices of amultidimensional array. Thus, the expression
foo["A", "B"]
really accesses
foo["A\034B"]
(see Multi-dimensional).
TEXTDOMAIN #
dcgettext()
,
dcngettext()
and
bindtextdomain()
functions(see Internationalization). The default value of
TEXTDOMAIN
is
"messages"
.
This variable is a gawk extension. In otherawk implementations,or if gawk is in compatibility mode(seeOptions),it is not special.
The following is an alphabetical list of variables thatawksets automatically on certain occasions in order to provideinformation to your program. The variables that are specific togawk are marked with a pound sign (‘#’).
ARGC
,
ARGV
ARGV
.
ARGC
is the number of command-linearguments present. See Other Arguments. Unlike most
awk arrays,
ARGV
is indexed from 0 to
ARGC
− 1. In the following example:
$ awk 'BEGIN { > for (i = 0; i < ARGC; i++) > print ARGV[i] > }' inventory-shipped BBS-list -| awk -| inventory-shipped -| BBS-list
ARGV[0]
contains ‘awk’,ARGV[1]
contains ‘inventory-shipped’, andARGV[2]
contains‘BBS-list’. The value ofARGC
is three, one more than theindex of the last element in ARGV
, because the elements are numberedfrom zero.
The namesARGC
and ARGV
, as well as the convention of indexingthe array from 0 toARGC
− 1, are derived from the C language'smethod of accessing command-line arguments.
The value ofARGV[0]
can vary from system to system. Also, you should note that the program text isnot included inARGV
, nor are any of awk's command-line options. SeeARGC and ARGV, for informationabout how awk uses these variables. (d.c.)
ARGIND #
ARGV
of the current file being processed. Every time
gawk opens a new data file for processing, it sets
ARGIND
to the index in
ARGV
of the file name. When
gawk is processing the input files,‘
FILENAME == ARGV[ARGIND]’ is always true.
This variable is useful in file processing; it allows you to tell how faralong you are in the list of data files as well as to distinguish betweensuccessive instances of the same file name on the command line.
While you can change the value ofARGIND
within your awkprogram,gawk automatically sets it to a new value when thenext file is opened.
This variable is a gawk extension. In otherawk implementations,or if gawk is in compatibility mode(seeOptions),it is not special.
ENVIRON
ENVIRON["HOME"]
might be
/home/arnold. Changing this arraydoes not affect the environment passed on to any programs that
awk may spawn via redirection or the
system()
function.
Some operating systems may not have environment variables. On such systems, theENVIRON
array is empty (except forENVIRON["AWKPATH"]
,seeAWKPATH Variable).
ERRNO #
getline
,during a read for
getline
, or during a
close()
operation,then
ERRNO
contains a string describing the error.
In addition, gawk clears ERRNO
before opening each command-line input file. This enables checking ifthe file is readable inside aBEGINFILE
pattern (see BEGINFILE/ENDFILE).
Otherwise,ERRNO
works similarly to the C variable errno
. Except for the case just mentioned,gawknever clears it (sets itto zero or ""
). Thus, you should only expect its valueto be meaningful when an I/O operation returns a failurevalue, such asgetline
returning −1. You are, of course, free to clear it yourself before doing anI/O operation.
This variable is a gawk extension. In otherawk implementations,or if gawk is in compatibility mode(seeOptions),it is not special.
FILENAME
FILENAME
is set to
"-"
.
FILENAME
is changed each time a new file is read(see Reading Files). Inside a
BEGIN
rule, the value of
FILENAME
is
""
, since there are no input files being processedyet.
36(d.c.) Note, though, that using
getline
(see Getline)inside a
BEGIN
rule can give
FILENAME
a value.
FNR
FNR
isincremented each time a new record is read(see Records). It is reinitializedto zero each time a new input file is started.
NF
NF
is set each time a new record is read, when a new field iscreated or when
$0
changes (see Fields).
Unlike most of the variables described in thissection,assigning a value to NF
has the potential to affectawk's internal workings. In particular, assignmentstoNF
can be used to create or remove fields from thecurrent record. SeeChanging Fields.
NR
NR
is incremented each time a new record is read.
PROCINFO #
PROCINFO["egid"]
getegid()
system call.
PROCINFO["euid"]
geteuid()
system call.
PROCINFO["FS"]
"FS"
if field splitting with
FS
is in effect,
"FIELDWIDTHS"
if field splitting with
FIELDWIDTHS
is in effect,or
"FPAT"
if field matching with
FPAT
is in effect.
PROCINFO["gid"]
getgid()
system call.
PROCINFO["pgrpid"]
PROCINFO["pid"]
PROCINFO["ppid"]
PROCINFO["sorted_in"]
PROCINFO
, its value controls theorder in which array indices will be processed by‘
for (index in array) ...’ loops. Since this is an advanced feature, we defer thefull description until later; see Scanning an Array.
PROCINFO["strftime"]
strftime()
. Assigning a new value to this element changes the default. See Time Functions.
PROCINFO["uid"]
getuid()
system call.
PROCINFO["version"]
On some systems, there may be elements in the array, "group1"
through"group
N"
for some N. N is the number ofsupplementary groups that the process has. Use thein
operatorto test for these elements(see Reference to Elements).
ThePROCINFO
array is also used to cause coprocessesto communicate over pseudo-ttys instead of through two-way pipes;this is discussed further inTwo-way I/O.
This array is a gawk extension. In otherawk implementations,or if gawk is in compatibility mode(seeOptions),it is not special.
RLENGTH
match()
function(see String Functions).
RLENGTH
is set by invoking the
match()
function. Its valueis the length of the matched string, or −1 if no match is found.
RSTART
match()
function(see String Functions).
RSTART
is set by invoking the
match()
function. Its valueis the position of the string where the matched substring starts, or zeroif no match was found.
RT #
RS
, the record separator.
This variable is a gawk extension. In otherawk implementations,or if gawk is in compatibility mode(seeOptions),it is not special.
NR
and FNR
awk increments NR
and FNR
each time it reads a record, instead of setting them to the absolutevalue of the number of records read. This means that a program canchange these variables and their new values are incremented foreach record. (d.c.) The following example shows this:
$ echo '1 > 2 > 3 > 4' | awk 'NR == 2 { NR = 17 } > { print NR }' -| 1 -| 17 -| 18 -| 19
Before FNR
was added to the awk language(seeV7/SVR3.1),many awk programs used this feature to track the number ofrecords in a file by resettingNR
to zero when FILENAME
changed.
ARGC
and ARGV
Auto-set,presented the following program describing the information contained in ARGC
and ARGV
:
$ awk 'BEGIN { > for (i = 0; i < ARGC; i++) > print ARGV[i] > }' inventory-shipped BBS-list -| awk -| inventory-shipped -| BBS-list
In this example, ARGV[0]
contains ‘awk’,ARGV[1]
contains ‘inventory-shipped’, andARGV[2]
contains‘BBS-list’. Notice that theawk program is not entered in ARGV
. Theother command-line options, with their arguments, are also notentered. This includes variable assignments done with the-voption (see Options). Normal variable assignments on the command linearetreated as arguments and do show up in the ARGV
array. Given the following program in a file namedshowargs.awk:
BEGIN { printf "A=%d, B=%d\n", A, B for (i = 0; i < ARGC; i++) printf "\tARGV[%d] = %s\n", i, ARGV[i] } END { printf "A=%d, B=%d\n", A, B }
Running it produces the following:
$ awk -v A=1 -f showargs.awk B=2 /dev/null -| A=1, B=0 -| ARGV[0] = awk -| ARGV[1] = B=2 -| ARGV[2] = /dev/null -| A=1, B=2
A program can alter ARGC
and the elements of ARGV
. Each timeawk reaches the end of an input file, it uses the nextelement ofARGV
as the name of the next input file. By storing adifferent string there, a program can change which files are read. Use"-"
to represent the standard input. Storingadditional elements and incrementingARGC
causesadditional files to be read.
If the value of ARGC
is decreased, that eliminates input filesfrom the end of the list. By recording the old value ofARGC
elsewhere, a program can treat the eliminated arguments assomething other than file names.
To eliminate a file from the middle of the list, store the null string(""
) intoARGV
in place of the file's name. As aspecial feature, awk ignores file names that have beenreplaced with the null string. Another option is touse thedelete
statement to remove elements fromARGV
(see Delete).
All of these actions are typically done in the BEGIN
rule,before actual processing of the input begins. SeeSplit Program, and seeTee Program, for examplesof each way of removing elements fromARGV
. The following fragment processes ARGV
in order to examine, andthen remove, command-line options:
BEGIN { for (i = 1; i < ARGC; i++) { if (ARGV[i] == "-v") verbose = 1 else if (ARGV[i] == "-q") debug = 1 else if (ARGV[i] ~ /^-./) { e = sprintf("%s: unrecognized option -- %c", ARGV[0], substr(ARGV[i], 2, 1)) print e > "/dev/stderr" } else break delete ARGV[i] } }
To actually get the options into the awk program,end theawk options with -- and then supplytheawk program's options, in the following manner:
awk -f myprog -- -v -q file1 file2 ...
This is not necessary ingawk. Unless --posix hasbeen specified,gawk silently puts any unrecognized optionsintoARGV
for the awk program to deal with. As soonas it sees an unknown option,gawk stops looking for otheroptions that it might otherwise recognize. The previous example withgawk would be:
gawk -f myprog -q -v file1 file2 ...
Because -q is not a validgawk option,it and the following -vare passed on to the awk program. (SeeGetopt Function, for an awk library functionthat parses command-line options.)
An array is a table of values calledelements. Theelements of an array are distinguished by their indices. Indicesmay be either numbers or strings.
This chapter describes how arrays work in awk,how to use array elements, how to scan through every element in an array,and how to remove array elements. It also describes howawk simulates multidimensionalarrays, as well as some of the less obvious points about array usage. The chapter moves on to discussgawk's facilityfor sorting arrays, and ends with a brief description ofgawk'sability to support true multidimensional arrays.
awk maintains a single setof names that may be used for naming variables, arrays, and functions(seeUser-defined). Thus, you cannot have a variable and an array with the same name in thesameawk program.
This section presents the basics: working with elementsin arrays one at a time, and traversing all of the elements inan array.
Doing linear scans over an associative array is like trying to club someoneto death with a loaded Uzi.
Larry Wall
The awk language provides one-dimensional arraysfor storing groups of related strings or numbers. Everyawk array must have a name. Array names have the samesyntax as variable names; any valid variable name would also be a validarray name. But one name cannot be used in both ways (as an array andas a variable) in the same awk program.
Arrays in awk superficially resemble arrays in other programminglanguages, but there are fundamental differences. Inawk, itisn't necessary to specify the size of an array before starting to use it. Additionally, any number or string inawk, not just consecutive integers,may be used as an array index.
In most other languages, arrays must be declared before use,including a specification ofhow many elements or components they contain. In such languages, thedeclaration causes a contiguous block of memory to be allocated for thatmany elements. Usually, an index in the array must be a positive integer. For example, the index zero specifies the first element in the array, which isactually stored at the beginning of the block of memory. Index onespecifies the second element, which is stored in memory right after thefirst element, and so on. It is impossible to add more elements to thearray, because it has room only for as many elements as given inthe declaration. (Some languages allow arbitrary starting and endingindices—e.g., ‘15 .. 27’—but the size of the array is still fixed whenthe array is declared.)
A contiguous array of four elements might look like the following example,conceptually, if the element values are 8,"foo"
,""
, and 30:
+---------+---------+--------+---------+ | 8 | "foo" | "" | 30 | Value +---------+---------+--------+---------+ 0 1 2 3 Index
Only the values are stored; the indices are implicit from the order ofthe values. Here, 8 is the value at index zero, because 8 appears in theposition with zero elements before it.
Arrays inawk are different—they are associative. This meansthat each array is a collection of pairs: an index and its correspondingarray element value:
Index 3 Value 30 Index 1 Value "foo" Index 0 Value 8 Index 2 Value ""
The pairs are shown in jumbled order because their order is irrelevant.
One advantage of associative arrays is that new pairs can be addedat any time. For example, suppose a tenth element is added to the arraywhose value is"number ten"
. The result is:
Index 10 Value "number ten" Index 3 Value 30 Index 1 Value "foo" Index 0 Value 8 Index 2 Value ""
Now the array issparse, which just means some indices are missing. It has elements 0–3 and 10, but doesn't have elements 4, 5, 6, 7, 8, or 9.
Another consequence of associative arrays is that the indices don'thave to be positive integers. Any number, or even a string, can bean index. For example, the following is an array that translates words fromEnglish to French:
Index "dog" Value "chien" Index "cat" Value "chat" Index "one" Value "un" Index 1 Value "un"
Here we decided to translate the number one in both spelled-out andnumeric form—thus illustrating that a single array can have bothnumbers and strings as indices. In fact, array subscripts are always strings; this is discussedin more detail inNumeric Array Subscripts. Here, the number1
isn't double-quoted, since awkautomatically converts it to a string.
The value of IGNORECASE
has no effect upon array subscripting. The identical string value used to store an array element must be usedto retrieve it. Whenawk creates an array (e.g., with the split()
built-in function),that array's indices are consecutive integers starting at one. (SeeString Functions.)
awk's arrays are efficient—the time to access an elementis independent of the number of elements in the array.
The principal way to use an array is to refer to one of its elements. An array reference is an expression as follows:
array[index-expression]
Here, array is the name of an array. The expressionindex-expression isthe index of the desired element of the array.
The value of the array reference is the current value of that arrayelement. For example,foo[4.3]
is an expression for the elementof array foo
at index ‘4.3’.
A reference to an array element that has no recorded value yields a value of""
, the null string. This includes elementsthat have not been assigned any value as well as elements that have beendeleted (seeDelete).
NOTE: A reference to an element that does not exist automatically createsthat array element, with the null string as its value. (In some cases,this is unfortunate, because it might waste memory inside awk.)Novice awk programmers often make the mistake of checking ifan element exists by checking if the value is empty:
# Check if "foo" exists in a: Incorrect! if (a["foo"] != "") ...This is incorrect, since this will create
a["foo"]
if it didn't exist before!
To determine whether an element exists in an array at a certain index, usethe following expression:
ind in array
This expression tests whether the particular indexind exists,without the side effect of creating that element if it is not present. The expression has the value one (true) ifarray[
ind]
exists and zero (false) if it does not exist. For example, this statement tests whether the arrayfrequencies
contains the index ‘2’:
if (2 in frequencies) print "Subscript 2 is present."
Note that this is not a test of whether the arrayfrequencies
contains an element whosevalue is two. There is no way to do that except to scan all the elements. Also, thisdoes not createfrequencies[2]
, while the following(incorrect) alternative does:
if (frequencies[2] != "") print "Subscript 2 is present."
Array elements can be assigned values just likeawk variables:
array[index-expression] = value
array is the name of an array. The expressionindex-expression is the index of the element of the array that isassigned a value. The expressionvalue is the value toassign to that element of the array.
The following program takes a list of lines, each beginning with a linenumber, and prints them out in order of line number. The line numbersare not in order when they are first read—instead theyare scrambled. This program sorts the lines by making an array usingthe line numbers as subscripts. The program then prints out the linesin sorted order of their numbers. It is a very simple program and getsconfused upon encountering repeated numbers, gaps, or lines that don'tbegin with a number:
{ if ($1 > max) max = $1 arr[$1] = $0 } END { for (x = 1; x <= max; x++) print arr[x] }
The first rule keeps track of the largest line number seen so far;it also stores each line into the arrayarr
, at an index thatis the line's number. The second rule runs after all the input has been read, to print outall the lines. When this program is run with the following input:
5 I am the Five man 2 Who are you? The new number two! 4 . . . And four on the floor 1 Who is number one? 3 I three you.
Its output is:
1 Who is number one? 2 Who are you? The new number two! 3 I three you. 4 . . . And four on the floor 5 I am the Five man
If a line number is repeated, the last line with a given number overridesthe others. Gaps in the line numbers can be handled with an easy improvement to theprogram'sEND
rule, as follows:
END { for (x = 1; x <= max; x++) if (x in arr) print arr[x] }
In programs that use arrays, it is often necessary to use a loop thatexecutes once for each element of an array. In other languages, wherearrays are contiguous and indices are limited to positive integers,this is easy: all the valid indices can be found by counting fromthe lowest index up to the highest. This technique won't do the jobinawk, because any number or string can be an array index. Soawk has a special kind of for
statement for scanningan array:
for (var in array) body
This loop executesbody once for each index in array that theprogram has previously used, with the variablevar set to that index.
The following program uses this form of thefor
statement. Thefirst rule scans the input records and notes which words appear (atleast once) in the input, by storing a one into the arrayused
withthe word as index. The second rule scans the elements of used
tofind all the distinct words that appear in the input. It prints eachword that is more than 10 characters long and also prints the number ofsuch words. SeeString Functions,for more information on the built-in functionlength()
.
# Record a 1 for each word that is used at least once { for (i = 1; i <= NF; i++) used[$i] = 1 } # Find number of distinct words more than 10 characters long END { for (x in used) { if (length(x) > 10) { ++num_long_words print x } } print num_long_words, "words longer than 10 characters" }
See Word Sorting,for a more detailed example of this type.
The order in which elements of the array are accessed by this statementis determined by the internal arrangement of the array elements withinawk and normally cannot be controlled or changed. This can lead toproblems if new elements are added toarray by statements inthe loop body; it is not predictable whether thefor
loop willreach them. Similarly, changing var inside the loop may producestrange results. It is best to avoid such things.
As an extension, gawk makes it possible for you toloop over the elements of an array in order, based on the value ofPROCINFO["sorted_in"]
(seeAuto-set). This is an advanced feature, so discussion of it is delayeduntilControlling Array Traversal.
In addition, gawk provides built-in functions forsorting arrays; seeArray Sorting Functions.
delete
StatementTo remove an individual element of an array, use the delete
statement:
delete array[index-expression]
Once an array element has been deleted, any value the element oncehad is no longer available. It is as if the element had neverbeen referred to or been given a value. The following is an example of deleting elements in an array:
for (i in frequencies) delete frequencies[i]
This example removes all the elements from the array frequencies
. Once an element is deleted, a subsequentfor
statement to scan the arraydoes not report that element and the in
operator to check forthe presence of that element returns zero (i.e., false):
delete foo[4] if (4 in foo) print "This will never be printed"
It is important to note that deleting an element isnot thesame as assigning it a null value (the empty string, ""
). For example:
foo[4] = "" if (4 in foo) print "This is printed, even though foo[4] is empty"
It is not an error to delete an element that does not exist. However, if--lint is provided on the command line(seeOptions),gawk issues a warning message when an element thatis not in the array is deleted.
All the elements of an array may be deleted with a single statement(c.e.) by leaving off the subscript in thedelete
statement,as follows:
delete array
This ability is a gawk extension; it is not available incompatibility mode (seeOptions).
Using this version of the delete
statement is about three timesmore efficient than the equivalent loop that deletes each element oneat a time.
The following statement provides a portable but nonobvious way to clearout an array:37
split("", array)
Thesplit()
function(see String Functions)clears out the target array first. This call asks it to splitapart the null string. Because there is no data to split out, thefunction simply clears the array and then returns.
CAUTION: Deleting an array does not change its type; you cannotdelete an array and then use the array's name as a scalar(i.e., a regular variable). For example, the following does not work:a[1] = 3 delete a a = 3
An important aspect to remember about arrays is that array subscriptsare always strings. When a numeric value is used as a subscript,it is converted to a string value before being used for subscripting(seeConversion). This means that the value of the built-in variableCONVFMT
canaffect how your program accesses elements of an array. For example:
xyz = 12.153 data[xyz] = 1 CONVFMT = "%2.2f" if (xyz in data) printf "%s is in data\n", xyz else printf "%s is not in data\n", xyz
This prints ‘12.15 is not in data’. The first statement givesxyz
a numeric value. Assigning todata[xyz]
subscriptsdata
with the string value "12.153"
(using the default conversion value ofCONVFMT
, "%.6g"
). Thus, the array element data["12.153"]
is assigned the value one. The program then changesthe value ofCONVFMT
. The test ‘(xyz in data)’ generates a newstring value fromxyz
—this time "12.15"
—because the value ofCONVFMT
only allows two significant digits. This test fails,since"12.15"
is different from "12.153"
.
According to the rules for conversions(seeConversion), integervalues are always converted to strings as integers, no matter what thevalue ofCONVFMT
may happen to be. So the usual case ofthe following works:
for (i = 1; i <= maxsub; i++) do something with array[i]
The “integer values always convert to strings as integers” rulehas an additional consequence for array indexing. Octal and hexadecimal constants(seeNondecimal-numbers)are converted internally into numbers, and their original formis forgotten. This means, for example, thatarray[17]
,array[021]
,andarray[0x11]
all refer to the same element!
As with many things in awk, the majority of the timethings work as one would expect them to. But it is useful to have a preciseknowledge of the actual rules since they can sometimes have a subtleeffect on your programs.
Suppose it's necessary to write a programto print the input data in reverse order. A reasonable attempt to do so (with some testdata) might look like this:
$ echo 'line 1 > line 2 > line 3' | awk '{ l[lines] = $0; ++lines } > END { > for (i = lines-1; i >= 0; --i) > print l[i] > }' -| line 3 -| line 2
Unfortunately, the very first line of input data did not come out in theoutput!
Upon first glance, we would think that this program should have worked. The variablelines
is uninitialized, and uninitialized variables have the numeric value zero. So,awk should have printed the value of l[0]
.
The issue here is that subscripts for awk arrays arealwaysstrings. Uninitialized variables, when used as strings, have thevalue""
, not zero. Thus, ‘line 1’ ends up stored inl[""]
. The following version of the program works correctly:
{ l[lines++] = $0 } END { for (i = lines - 1; i >= 0; --i) print l[i] }
Here, the ‘++’ forces lines
to be numeric, thus makingthe “old value” numeric zero. This is then converted to"0"
as the array subscript.
Even though it is somewhat unusual, the null string(""
) is a valid array subscript. (d.c.) gawk warns about the use of the null string as a subscriptif--lint is providedon the command line (seeOptions).
A multidimensional array is an array in which an element is identifiedby a sequence of indices instead of a single index. For example, atwo-dimensional array requires two indices. The usual way (in mostlanguages, includingawk) to refer to an element of atwo-dimensional array namedgrid
is withgrid[
x,
y]
.
Multidimensional arrays are supported inawk throughconcatenation of indices into one string.awk converts the indices into strings(seeConversion) andconcatenates them together, with a separator between them. This createsa single string that describes the values of the separate indices. Thecombined string is used as a single index into an ordinary,one-dimensional array. The separator used is the value of the built-invariable SUBSEP
.
For example, suppose we evaluate the expression ‘foo[5,12] = "value"’when the value ofSUBSEP
is "@"
. The numbers 5 and 12 areconverted to strings andconcatenated with an ‘@’ between them, yielding"5@12"
; thus,the array element foo["5@12"]
is set to "value"
.
Once the element's value is stored, awk has no record of whetherit was stored with a single index or a sequence of indices. The twoexpressions ‘foo[5,12]’ and ‘foo[5 SUBSEP 12]’ are alwaysequivalent.
The default value of SUBSEP
is the string "\034"
,which contains a nonprinting character that is unlikely to appear in anawk program or in most input data. The usefulness of choosing an unlikely character comes from the factthat index values that contain a string matching SUBSEP
can lead tocombined strings that are ambiguous. Suppose that SUBSEP
is"@"
; then ‘foo["a@b", "c"]’ and ‘foo["a", "b@c"]’ are indistinguishable because both are actuallystored as ‘foo["a@b@c"]’.
To test whether a particular index sequence exists in amultidimensional array, use the same operator (in
) that isused for single dimensional arrays. Write the whole sequence of indicesin parentheses, separated by commas, as the left operand:
(subscript1, subscript2, ...) in array
The following example treats its input as a two-dimensional array offields; it rotates this array 90 degrees clockwise and prints theresult. It assumes that all lines have the same number ofelements:
{ if (max_nf < NF) max_nf = NF max_nr = NR for (x = 1; x <= NF; x++) vector[x, NR] = $x } END { for (x = 1; x <= max_nf; x++) { for (y = max_nr; y >= 1; --y) printf("%s ", vector[x, y]) printf("\n") } }
When given the input:
1 2 3 4 5 6 2 3 4 5 6 1 3 4 5 6 1 2 4 5 6 1 2 3
the program produces the following output:
4 3 2 1 5 4 3 2 6 5 4 3 1 6 5 4 2 1 6 5 3 2 1 6
There is no special for
statement for scanning a“multidimensional” array. There cannot be one, because, in truth, thereare no multidimensional arrays or elements—there is only amultidimensionalway of accessing an array.
However, if your program has an array that is always accessed asmultidimensional, you can get the effect of scanning it by combiningthe scanning for
statement(see Scanning an Array) with thebuilt-in split()
function(see String Functions). It works in the following manner:
for (combined in array) { split(combined, separate, SUBSEP) ... }
This sets the variable combined
toeach concatenated combined index in the array, and splits itinto the individual indices by breaking it apart where the value ofSUBSEP
appears. The individual indices then become the elements ofthe array separate
.
Thus, if a value is previously stored in array[1, "foo"]
; thenan element with index"1\034foo"
exists in array
. (Recallthat the default value ofSUBSEP
is the character with code 034.) Sooner or later, the for
statement finds that index and does aniteration with the variablecombined
set to "1\034foo"
. Then the split()
function is called as follows:
split("1\034foo", separate, "\034")
The result is to set separate[1]
to "1"
andseparate[2]
to"foo"
. Presto! The original sequence ofseparate indices is recovered.
gawk supports arrays ofarrays. Elements of a subarray are referred to by their own indicesenclosed in square brackets, just like the elements of the main array. For example, the following creates a two-element subarray at index ‘1’of the main array a
:
a[1][1] = 1 a[1][2] = 2
This simulates a true two-dimensional array. Each subarray element cancontain another subarray as a value, which in turn can hold other arraysas well. In this way, you can create arrays of three or more dimensions. The indices can be anyawk expression, including scalarsseparated by commas (that is, a regularawk simulatedmultidimensional subscript). So the following is valid ingawk:
a[1][3][1, "name"] = "barney"
Each subarray and the main array can be of different length. In fact, theelements of an array or its subarray do not all have to have the sametype. This means that the main array and any of its subarrays can benon-rectangular, or jagged in structure. One can assign a scalar value tothe index ‘4’ of the main arraya
:
a[4] = "An element in a jagged array"
The terms dimension, row and column aremeaningless when appliedto such an array, but we will use “dimension” henceforth to imply themaximum number of indices needed to refer to an existing element. Thetype of any element that has already been assigned cannot be changedby assigning a value of a different type. You have to first delete thecurrent element, which effectively makesgawk forget aboutthe element at that index:
delete a[4] a[4][5][6][7] = "An element in a four-dimensional array"
This removes the scalar value from index ‘4’ and then inserts asubarray of subarray of subarray containing a scalar. You can alsodelete an entire subarray or subarray of subarrays:
delete a[4][5] a[4][5] = "An element in subarray a[4]"
But recall that you can not delete the main array a
and then use itas a scalar.
The built-in functions which take array arguments can also be usedwith subarrays. For example, the following code fragment useslength()
(see String Functions)to determine the number of elements in the main arraya
andits subarrays:
print length(a), length(a[1]), length(a[1][3])
This results in the following output for our main array a
:
2, 3, 1
The ‘subscript in array’ expression(see Reference to Elements) works similarly for bothregularawk-stylearrays and arrays of arrays. For example, the tests ‘1 in a’,‘3 in a[1]’, and ‘(1, "name") in a[1][3]’ all evaluate toone (true) for our array a
.
The ‘for (item in array)’ statement (seeScanning an Array)can be nested to scan all theelements of an array of arrays if it is rectangular in structure. In orderto print the contents (scalar values) of a two-dimensional array of arrays(i.e., in which each first-level element is itself anarray, not necessarily of the same length)you could use the following code:
for (i in array) for (j in array[i]) print array[i][j]
The isarray()
function (see Type Functions)lets you test if an array element is itself an array:
for (i in array) { if (isarray(array[i]) { for (j in array[i]) { print array[i][j] } } }
If the structure of a jagged array of arrays is known in advance,you can often devise workarounds using control statements. For example,the following code prints the elements of our main arraya
:
for (i in a) { for (j in a[i]) { if (j == 3) { for (k in a[i][j]) print a[i][j][k] } else print a[i][j] } }
See Walking Arrays, for a user-defined function that will “walk” anarbitrarily-dimensioned array of arrays.
Recall that a reference to an uninitialized array element yields a valueof ""
, the null string. This has one important implication when youintend to use a subarray as an argument to a function, as illustrated bythe following example:
$ gawk 'BEGIN { split("a b c d", b[1]); print b[1][1] }' error--> gawk: cmd. line:1: fatal: split: second argument is not an array
The way to work around this is to first force b[1]
to be an array bycreating an arbitrary index:
$ gawk 'BEGIN { b[1][1] = ""; split("a b c d", b[1]); print b[1][1] }' -| a
This chapter describesawk's built-in functions,which fall into three categories: numeric, string, and I/O.gawk provides additional groups of functionsto work with values that represent time, dobit manipulation, sort arrays, and internationalize and localize programs.
Besides the built-in functions, awk has provisions forwriting new functions that the rest of a program can use. The second half of this chapter describes theseuser-defined functions.
Built-in functions are always available foryour awk program to call. This section defines allthe built-infunctions inawk; some of these are mentioned in other sectionsbut are summarized here for your convenience.
To call one of awk's built-in functions, write the name ofthe function followedby arguments in parentheses. For example, ‘atan2(y + z, 1)’is a call to the functionatan2()
and has two arguments.
Whitespace is ignored between the built-in function name and theopen parenthesis, but nonetheless it is good practice to avoid using whitespacethere. User-defined functions do not permit whitespace in this way, andit is easier to avoid mistakes by following a simpleconvention that always works—no whitespace after a function name.
Each built-in function accepts a certain number of arguments. In some cases, arguments can be omitted. The defaults for omittedarguments vary from function to function and are described under theindividual functions. In someawk implementations, extraarguments given to built-in functions are ignored. However, ingawk,it is a fatal error to give extra arguments to a built-in function.
When a function is called, expressions that create the function's actualparameters are evaluated completely before the call is performed. For example, in the following code fragment:
i = 4 j = sqrt(i++)
the variablei
is incremented to the value five before sqrt()
is called with a value of four for its actual parameter. The order of evaluation of the expressions used for the function'sparameters is undefined. Thus, avoid writing programs thatassume that parameters are evaluated from left to right or fromright to left. For example:
i = 5 j = atan2(i++, i *= 2)
If the order of evaluation is left to right, then i
first becomes6, and then 12, andatan2()
is called with the two arguments 6and 12. But if the order of evaluation is right to left,i
first becomes 10, then 11, and atan2()
is called with thetwo arguments 11 and 10.
The following list describes all ofthe built-in functions that work with numbers. Optional parameters are enclosed in square brackets ([ ]):
atan2(
y
,
x
)
/
x in radians.
cos(
x
)
exp(
x
)
e ^
x) or reportan error if x is out of range. The range of values x can havedepends on your machine's floating-point representation.
int(
x
)
For example, int(3)
is 3, int(3.9)
is 3, int(-3.9)
is −3, andint(-3)
is −3 as well.
log(
x
)
rand()
rand()
areuniformly distributed between zero and one. The value could be zero but is never one.
38
Often random integers are needed instead. Following is a user-defined functionthat can be used to obtain a random non-negative integer less thann:
function randint(n) { return int(n * rand()) }
The multiplication produces a random number greater than zero and lessthann
. Using int()
, this result is made intoan integer between zero andn
− 1, inclusive.
The following example uses a similar function to produce random integersbetween one andn. This program prints a new random number foreach input record:
# Function to roll a simulated die. function roll(n) { return 1 + int(rand() * n) } # Roll 3 six-sided dice and # print total number of points. { printf("%d points\n", roll(6)+roll(6)+roll(6)) }
CAUTION: In most awk implementations, including gawk,rand()
starts generating numbers from the samestarting number, or seed, each time you run awk. 39 Thus,a program generates the same results each time you run it. The numbers are random within one awk run but predictablefrom run to run. This is convenient for debugging, but if you wanta program to do different things each time it is used, you must changethe seed to a value that is different in each run. To do this,usesrand()
.
sin(
x
)
sqrt(
x
)
sqrt(4)
is 2.
srand(
[ x
]
)
Each seed value leads to a particular sequence of randomnumbers.40Thus, if the seed is set to the same value a second time,the same sequence of random numbers is produced again.
CAUTION: Different awk implementations use different random-numbergenerators internally. Don't expect the same awk programto produce the same series of random numbers when executed bydifferent versions of awk.
If the argument x is omitted, as in ‘srand()’, then the currentdate and time of day are used for a seed. This is the way to get randomnumbers that are truly unpredictable.
The return value of srand()
is the previous seed. This makes iteasy to keep track of the seeds in case you need to consistently reproducesequences of random numbers.
The functions in this section look at or change the text of one or morestrings.gawk
understands locales (see Locales), and does all string processing in terms ofcharacters, notbytes. This distinction is particularly importantto understand for locales where one charactermay be represented by multiple bytes. Thus, for example,length()
returns the number of characters in a string, and not the number of bytesused to represent those characters, Similarly,index()
works withcharacter indices, and not byte indices.
In the following list, optional parameters are enclosed in square brackets ([ ]).Several functions perform string substitution; the full discussion isprovided in the description of thesub()
function, which comestowards the end since the list is presented in alphabetic order. Those functions that are specific togawk are marked with apound sign (‘#’):
asort(
source
[
,
dest
[
,
how
]
]
) #
"ascending string"
for the value of how. If the ‘
source’ array contains subarrays as values,they will come out last(first) in the ‘
dest’ array for ‘
ascending’(‘
descending’)order specification. The value of
IGNORECASE
affects the sorting. The third argument can also be a user-defined function name in which casethe value returned by the function is used to order the array elementsbefore constructing the result array. See Array Sorting Functions, for more information.
For example, if the contents of a
are as follows:
a["last"] = "de" a["first"] = "sac" a["middle"] = "cul"
A call to asort()
:
asort(a)
results in the following contents of a
:
a[1] = "cul" a[2] = "de" a[3] = "sac"
In order to reverse the direction of the sorted results in the above example,asort()
can be called with three arguments as follows:
asort(a, a, "descending")
The asort()
function is described in more detail inArray Sorting Functions.asort()
is a gawk extension; it is not availablein compatibility mode (seeOptions).
asorti(
source
[
,
dest
[
,
how
]
]
) #
asort()
, however, the
indicesare sorted, instead of the values. (Here too,
IGNORECASE
affects the sorting.)
The asorti()
function is described in more detail inArray Sorting Functions.asorti()
is a gawk extension; it is not availablein compatibility mode (seeOptions).
gensub(
regexp
,
replacement
,
how
[
,
target
]
) #
$0
. It returns the modified string as the resultof the function and the original target string is
not changed.
gensub()
is a general substitution function. It's purpose isto provide more features than the standardsub()
and gsub()
functions.
gensub()
provides an additional feature that is not availablein sub()
or gsub()
: the ability to specify components of aregexp in the replacement text. This is done by using parentheses inthe regexp to mark the components and then specifying ‘\N’in the replacement text, where N is a digit from 1 to 9. For example:
$ gawk ' > BEGIN { > a = "abc def" > b = gensub(/(.+) (.+)/, "\\2 \\1", "g", a) > print b > }' -| def abc
As with sub()
, you must type two backslashes in orderto get one into the string. In the replacement text, the sequence ‘\0’ represents the entirematched text, as does the character ‘&’.
The following example shows how you can use the third argument to controlwhich match of the regexp should be changed:
$ echo a b c a b c | > gawk '{ print gensub(/a/, "AA", 2) }' -| a b c AA b c
In this case, $0
is the default target string. gensub()
returns the new string as its result, which ispassed directly toprint
for printing.
If the how argument is a string that does not begin with ‘g’ or‘G’, or if it is a number that is less than or equal to zero, only onesubstitution is performed. Ifhow is zero, gawk issuesa warning message.
If regexp does not match target, gensub()
's return valueis the original unchanged value oftarget.
gensub()
is a gawk extension; it is not availablein compatibility mode (seeOptions).
gsub(
regexp
,
replacement
[
,
target
]
)
gsub()
stands for“global,” which means replace everywhere. For example:
{ gsub(/Britain/, "United Kingdom"); print }
replaces all occurrences of the string ‘Britain’ with ‘UnitedKingdom’ for all input records.
The gsub()
function returns the number of substitutions made. Ifthe variable to search and alter (target) isomitted, then the entire input record ($0
) is used. As insub()
, the characters ‘&’ and ‘\’ are special,and the third argument must be assignable.
index(
in
,
find
)
$ awk 'BEGIN { print index("peanut", "an") }' -| 3
If find is not found, index()
returns zero. (Remember that string indices inawk start at one.)
length(
[ string
]
)
length("abcde")
is five. Bycontrast,
length(15 * 35)
works out to three. In this example, 15 * 35 =525, and 525 is then converted to the string
"525"
, which hasthree characters.
If no argument is supplied, length()
returns the length of $0
.
NOTE: In older versions of
awk, the
length()
function couldbe calledwithout any parentheses. Doing so is considered poor practice,although the 2008 POSIX standard explicitly allows it, tosupport historical practice. For programs to be maximally portable,always supply the parentheses.
Iflength()
is called with a variable that has not been used,gawk forces the variable to be a scalar. Otherimplementations ofawk leave the variable without a type. (d.c.) Consider:
$ gawk 'BEGIN { print length(x) ; x[1] = 1 }' -| 0 error--> gawk: fatal: attempt to use scalar `x' as array $ nawk 'BEGIN { print length(x) ; x[1] = 1 }' -| 0
If --lint hasbeen specified on the command line,gawk issues awarning about this.
Withgawk and several other awk implementations, when given anarray argument, thelength()
function returns the number of elementsin the array. (c.e.) This is less useful than it might seem at first, as thearray is not guaranteed to be indexed from one to the number of elementsin it. If--lint is provided on the command line(seeOptions),gawk warns that passing an array argument is not portable. If--posix is supplied, using an array argument is a fatal error(seeArrays).
match(
string
,
regexp
[
,
array
]
)
The regexp argument may be either a regexp constant(/.../
) or a string constant ("..."
). In the latter case, the string is treated as a regexp to be matched. SeeComputed Regexps, for adiscussion of the difference between the two forms, and theimplications for writing your program correctly.
The order of the first two arguments is backwards from most other stringfunctions that work with regular expressions, such assub()
andgsub()
. It might help to remember thatfor match()
, the order is the same as for the ‘~’ operator:‘string ~regexp’.
Thematch()
function sets the built-in variable RSTART
tothe index. It also sets the built-in variableRLENGTH
to thelength in characters of the matched substring. If no match is found,RSTART
is set to zero, andRLENGTH
to −1.
For example:
{ if ($1 == "FIND") regex = $2 else { where = match($0, regex) if (where != 0) print "Match of", regex, "found at", where, "in", $0 } }
This program looks for lines that match the regular expression stored inthe variableregex
. This regular expression can be changed. If thefirst word on a line is ‘FIND’,regex
is changed to be thesecond word on that line. Therefore, if given:
FIND ru+n My program runs but not very quickly FIND Melvin JF+KM This line is property of Reality Engineering Co. Melvin was here.
awk prints:
Match of ru+n found at 12 in My program runs Match of Melvin found at 1 in Melvin was here.
Ifarray is present, it is cleared, and then the zeroth elementof array is set to the entire portion ofstringmatched by regexp. If regexp contains parentheses,the integer-indexed elements ofarray are set to contain theportion of string matching the corresponding parenthesizedsubexpression. For example:
$ echo foooobazbarrrrr | > gawk '{ match($0, /(fo+).+(bar*)/, arr) > print arr[1], arr[2] }' -| foooo barrrrr
In addition,multidimensional subscripts are available providingthe start index and length of each matched subexpression:
$ echo foooobazbarrrrr | > gawk '{ match($0, /(fo+).+(bar*)/, arr) > print arr[1], arr[2] > print arr[1, "start"], arr[1, "length"] > print arr[2, "start"], arr[2, "length"] > }' -| foooo barrrrr -| 1 5 -| 9 7
There may not be subscripts for the start and index for every parenthesizedsubexpression, since they may not all have matched text; thus theyshould be tested for with thein
operator(see Reference to Elements).
Thearray argument to match()
is agawk extension. In compatibility mode(seeOptions),using a third argument is a fatal error.
patsplit(
string
,
array
[
,
fieldpat
[
,
seps
]
]
) #
[1]
, the second piece in array
[2]
, and soforth. The third argument, fieldpat, isa regexp describing the fields in string (just as
FPAT
isa regexp describing the fields in input records). It may be either a regexp constant or a string. If fieldpat is omitted, the value of
FPAT
is used.
patsplit()
returns the number of elements created. seps
[
i
]
isthe separator stringbetween array
[
i
]
and array
[
i
+1]
. Any leading separator will be in seps
[0]
.
The patsplit()
function splits strings into pieces in amanner similar to the way input lines are split into fields usingFPAT
(see Splitting By Content.
Before splitting the string, patsplit()
deletes any previously existingelements in the arraysarray and seps.
Thepatsplit()
function is agawk extension. In compatibility mode(seeOptions),it is not available.
split(
string
,
array
[
,
fieldsep
[
,
seps
]
]
)
[1]
, the second piece in array
[2]
, and soforth. The string value of the third argument, fieldsep, isa regexp describing where to split string (much as
FS
canbe a regexp describing where to split input records;see Regexp Field Splitting). If fieldsep is omitted, the value of
FS
is used.
split()
returns the number of elements created. seps is a
gawk extension with seps
[
i
]
being the separator stringbetween array
[
i
]
and array
[
i
+1]
. If fieldsep is a singlespace then any leading whitespace goes into seps
[0]
andany trailingwhitespace goes into seps
[
n
]
where n is thereturn value of
split()
(that is, the number of elements in array).
The split()
function splits strings into pieces in amanner similar to the way input lines are split into fields. For example:
split("cul-de-sac", a, "-", seps)
splits the string ‘cul-de-sac’ into three fields using ‘-’ as theseparator. It sets the contents of the arraya
as follows:
a[1] = "cul" a[2] = "de" a[3] = "sac"
and sets the contents of the array seps
as follows:
seps[1] = "-" seps[2] = "-"
The value returned by this call to split()
is three.
As with input field-splitting, when the value offieldsep is" "
, leading and trailing whitespace is ignored in values assigned tothe elements ofarray but not inseps, and the elementsare separated by runs of whitespace. Also as with input field-splitting, iffieldsep is the null string, eachindividual character in the string is split into its own array element. (c.e.)
Note, however, that RS
has no effect on the way split()
works. Even though ‘RS = ""’ causes newline to also be an inputfield separator, this does not affect howsplit()
splits strings.
Modern implementations ofawk, including gawk, allowthe third argument to be a regexp constant (/abc/
) as well as astring. (d.c.) The POSIX standard allows this as well. SeeComputed Regexps, for adiscussion of the difference between using a string constant or a regexp constant,and the implications for writing your program correctly.
Before splitting the string, split()
deletes any previously existingelements in the arraysarray and seps.
If string is null, the array has no elements. (So this is a portableway to delete an entire array with one statement. SeeDelete.)
If string does not match fieldsep at all (but is not null),array has one element only. The value of that element is the originalstring.
sprintf(
format
,
expression1
, ...)
printf
wouldhave printed out with the same arguments(see Printf). For example:
pival = sprintf("pi = %.2f (approx.)", 22/7)
assigns the string ‘pi = 3.14 (approx.)’ to the variablepival
.
strtonum(
str
) #
strtonum()
assumes that stris an octal number. If str begins with a leading ‘
0x’ or‘
0X’,
strtonum()
assumes that str is a hexadecimal number. For example:
$ echo 0x11 | > gawk '{ printf "%d\n", strtonum($1) }' -| 17
Using the strtonum()
function is not the same as adding zeroto a string value; the automatic coercion of strings to numbersworks only for decimal data, not for octal or hexadecimal.41
Note also that strtonum()
uses the current locale's decimal pointfor recognizing numbers (seeLocales).
strtonum()
is agawk extension; it is not availablein compatibility mode (seeOptions).
sub(
regexp
,
replacement
[
,
target
]
)
The regexp argument may be either a regexp constant(/.../
) or a string constant ("..."
). In the latter case, the string is treated as a regexp to be matched. SeeComputed Regexps, for adiscussion of the difference between the two forms, and theimplications for writing your program correctly.
This function is peculiar because target is not simplyused to compute a value, and not just any expression will do—itmust be a variable, field, or array element so thatsub()
canstore a modified value there. If this argument is omitted, then thedefault is to use and alter$0
.42For example:
str = "water, water, everywhere" sub(/at/, "ith", str)
sets str
to ‘wither, water, everywhere’, by replacing theleftmost longest occurrence of ‘at’ with ‘ith’.
If the special character ‘&’ appears inreplacement, itstands for the precise substring that was matched by regexp. (Ifthe regexp can match more than one string, then this precise substringmay vary.) For example:
{ sub(/candidate/, "& and his wife"); print }
changes the first occurrence of ‘candidate’ to ‘candidateand his wife’ on each input line. Here is another example:
$ awk 'BEGIN { > str = "daabaaa" > sub(/a+/, "C&C", str) > print str > }' -| dCaaCbaaa
This shows how ‘&’ can represent a nonconstant string and alsoillustrates the “leftmost, longest” rule in regexp matching(seeLeftmost Longest).
The effect of this special character (‘&’) can be turned off by putting abackslash before it in the string. As usual, to insert one backslash inthe string, you must write two backslashes. Therefore, write ‘\\&’in a string constant to include a literal ‘&’ in the replacement. For example, the following shows how to replace the first ‘|’ on each line withan ‘&’:
{ sub(/\|/, "\\&"); print }
As mentioned, the third argument tosub()
mustbe a variable, field or array element. Some versions of awk allow the third argument tobe an expression that is not an lvalue. In such a case,sub()
still searches for the pattern and returns zero or one, but the result ofthe substitution (if any) is thrown away because there is no placeto put it. Such versions ofawk accept expressionslike the following:
sub(/USA/, "United States", "the USA and Canada")
For historical compatibility,gawk accepts such erroneous code. However, using any other nonchangeableobject as the third parameter causes a fatal error and your programwill not run.
Finally, if the regexp is not a regexp constant, it is converted into astring, and then the value of that string is treated as the regexp to match.
substr(
string
,
start
[
,
length
]
)
substr("washington", 5, 3)
returns
"ing"
.
If length is not present, substr()
returns the whole suffix ofstring that begins at character numberstart. For example,substr("washington", 5)
returns "ington"
. The wholesuffix is also returnedif length is greater than the number of characters remainingin the string, counting from characterstart.
If start is less than one, substr()
treats it asif it was one. (POSIX doesn't specify what to do in this case:Brian Kernighan'sawk acts this way, and therefore gawkdoes too.) If start is greater than the number of charactersin the string,substr()
returns the null string. Similarly, if length is present but less than or equal to zero,the null string is returned.
The string returned bysubstr()
cannot beassigned. Thus, it is a mistake to attempt to change a portion ofa string, as shown in the following example:
string = "abcdef" # try to get "abCDEf", won't work substr(string, 3, 3) = "CDE"
It is also a mistake to use substr()
as the third argumentofsub()
or gsub()
:
gsub(/xyz/, "pdq", substr($0, 5, 20)) # WRONG
(Some commercial versions ofawk treatsubstr()
as assignable, but doing so is not portable.)
If you need to replace bits and pieces of a string, combine substr()
with string concatenation, in the following manner:
string = "abcdef" ... string = substr(string, 1, 2) "CDE" substr(string, 6)
tolower(
string
)
tolower("MiXeD cAsE 123")
returns
"mixed case 123"
.
toupper(
string
)
toupper("MiXeD cAsE 123")
returns
"MIXED CASE 123"
.
sub()
, gsub()
, and gensub()
When using sub()
, gsub()
, or gensub()
, and trying to get literalbackslashes and ampersands into the replacement text, you need to rememberthat there are several levels ofescape processing going on.
First, there is the lexical level, which is when awk readsyour programand builds an internal copy of it that can be executed. Then there is the runtime level, which is whenawk actually scans thereplacement string to determine what to generate.
At both levels, awk looks for a defined set of characters thatcan come after a backslash. At the lexical level, it looks for theescape sequences listed inEscape Sequences. Thus, for every ‘\’ thatawk processes at the runtimelevel, you must type two backslashes at the lexical level. When a character that is not valid for an escape sequence follows the‘\’, Brian Kernighan'sawk and gawk both simply remove the initial‘\’ and put the next character into the string. Thus, forexample,"a\qb"
is treated as "aqb"
.
At the runtime level, the various functions handle sequences of‘\’ and ‘&’ differently. The situation is (sadly) somewhat complex. Historically, thesub()
and gsub()
functions treated the twocharacter sequence ‘\&’ specially; this sequence was replaced inthe generated text with a single ‘&’. Any other ‘\’ withinthe replacement string that did not precede an ‘&’ was passedthrough unchanged. This is illustrated intable-sub-escapes.
You typesub()
seessub()
generates ———– ————– ———————\&
&
the matched text\\&
\&
a literal ‘&’\\\&
\&
a literal ‘&’\\\\&
\\&
a literal ‘\&’\\\\\&
\\&
a literal ‘\&’\\\\\\&
\\\&
a literal ‘\\&’\\q
\q
a literal ‘\q’
Table 9.1: Historical Escape Sequence Processing forsub()
and gsub()
This table shows both the lexical-level processing, wherean odd number of backslashes becomes an even number at the runtime level,as well as the runtime processing done bysub()
. (For the sake of simplicity, the rest of the following tables only show thecase of even numbers of backslashes entered at the lexical level.)
The problem with the historical approach is that there is no way to geta literal ‘\’ followed by the matched text.
The POSIX rules state that ‘\&’ in the replacement string producesa literal ‘&’, ‘\\’ produces a literal ‘\’, and ‘\’ followedby anything else is not special; the ‘\’ is placed straight into the output. These rules are presented intable-posix-sub.
You typesub()
seessub()
generates ———– ————– ———————\\\\\\&
\\\&
a literal ‘\&’\\\\&
\\&
a literal ‘\’, followed by the matched text\\&
\&
a literal ‘&’\\q
\q
a literal ‘\q’\\\\
\\
\
Table 9.2: POSIX rules for sub()
andgsub()
gawk follows the POSIX rules.
The rules for gensub()
are considerably simpler. At the runtimelevel, whenevergawk sees a ‘\’, if the following characteris a digit, then the text that matched the corresponding parenthesizedsubexpression is placed in the generated output. Otherwise,no matter what character follows the ‘\’, itappears in the generated text and the ‘\’ does not,as shown intable-gensub-escapes.
You typegensub()
seesgensub()
generates ———– —————— ————————–&
&
the matched text\\&
\&
a literal ‘&’\\\\
\\
a literal ‘\’\\\\&
\\&
a literal ‘\’, then the matched text\\\\\\&
\\\&
a literal ‘\&’\\q
\q
a literal ‘q’
Table 9.3: Escape Sequence Processing for gensub()
Because of the complexity of the lexical and runtime level processingand the special cases forsub()
and gsub()
,we recommend the use of gawk andgensub()
when you haveto do substitutions.
Inawk, the ‘*’ operator can match the null string. This is particularly important for thesub()
, gsub()
,and gensub()
functions. For example:
$ echo abc | awk '{ gsub(/m*/, "X"); print }' -| XaXbXcX
Although this makes a certain amount of sense, it can be surprising.
The following functions relate to input/output (I/O). Optional parameters are enclosed in square brackets ([ ]):
close(
filename
[
,
how
]
)
When closing a coprocess, it is occasionally useful to first closeone end of the two-way pipe and then to close the other. This is doneby providing a second argument toclose()
. This second argumentshould be one of the two string values "to"
or "from"
,indicating which end of the pipe to close. Case in the string doesnot matter. SeeTwo-way I/O,which discusses this feature in more detail and gives an example.
fflush(
[ filename
]
)
Many utility programsbuffer their output; i.e., they save informationto write to a disk file or the screen in memory until there is enoughfor it to be worthwhile to send the data to the output device. This is often more efficient than writingevery little bit of information as soon as it is ready. However, sometimesit is necessary to force a program to flush its buffers; that is,write the information to its destination, even if a buffer is not full. This is the purpose of thefflush()
function—gawk alsobuffers its output and thefflush()
function forcesgawk to flush its buffers.
fflush()
was added to Brian Kernighan'sversion of awk in 1994; it is not part of the POSIX standard and isnot available if--posix has been specified on thecommand line (seeOptions).
gawk extends thefflush()
function in two ways. The firstis to allow no argument at all. In this case, the buffer for thestandard output is flushed. The second is to allow the null string(""
) as the argument. In this case, the buffers forall open output files and pipes are flushed. Brian Kernighan's awk also supports these extensions.
fflush()
returns zero if the buffer is successfully flushed;otherwise, it returns −1. In the case where all buffers are flushed, the return value is zeroonly if all buffers were flushed successfully. Otherwise, it is−1, and gawk warns about the problemfilename.
gawk also issues a warning message if you attempt to flusha file or pipe that was opened for reading (such as withgetline
),or if filename is not an open file, pipe, or coprocess. In such a case,fflush()
returns −1, as well.
system(
command
)
For example, if the following fragment of code is put in your awkprogram:
END { system("date | mail -s 'awk run done' root") }
the system administrator is sent mail when the awk programfinishes processing input and begins its end-of-input processing.
Note that redirecting print
or printf
into a pipe is oftenenough to accomplish your task. If you need to run many commands, itis more efficient to simply print them down a pipeline to the shell:
while (more stuff to do) print command | "/bin/sh" close("/bin/sh")
However, if yourawkprogram is interactive, system()
is useful for running largeself-contained programs, such as a shell or an editor. Some operating systems cannot implement thesystem()
function. system()
causes a fatal error if it is not supported.
NOTE: When
--sandbox is specified, the
system()
function is disabled(see Options).
As a side point, buffering issues can be even more confusing, dependingupon whether your program isinteractive, i.e., communicatingwith a user sitting at a keyboard.44
Interactive programs generally line buffer their output; i.e., theywrite out every line. Noninteractive programs wait until they havea full buffer, which may be many lines of output. Here is an example of the difference:
$ awk '{ print $1 + $2 }' 1 1 -| 2 2 3 -| 5 Ctrl-d
Each line of output is printed immediately. Compare that behaviorwith this example:
$ awk '{ print $1 + $2 }' | cat 1 1 2 3 Ctrl-d -| 2 -| 5
Here, no output is printed until after the Ctrl-d is typed, becauseit is all buffered and sent down the pipe tocat in one shot.
system()
Thefflush()
function provides explicit control over output buffering forindividual files and pipes. However, its use is not portable to many otherawk implementations. An alternative method to flush outputbuffers is to call system()
with a null string as its argument:
system("") # flush output
gawk treats this use of thesystem()
function as a specialcase and is smart enough not to run a shell (or other commandinterpreter) with the empty command. Therefore, withgawk, thisidiom is not only useful, it is also efficient. While this method should workwith otherawk implementations, it does not necessarily avoidstarting an unnecessary shell. (Other implementations may onlyflush the buffer associated with the standard output and not necessarilyall buffered output.)
If you think about what a programmer expects, it makes sense thatsystem()
should flush any pending output. The following program:
BEGIN { print "first print" system("echo system echo") print "second print" }
must print:
first print system echo second print
and not:
system echo first print second print
If awk did not flush its buffers before callingsystem()
,you would see the latter (undesirable) output.
awk
programs are commonly used to process log filescontaining timestamp information, indicating when aparticular log record was written. Many programs log their timestampin the form returned by thetime()
system call, which is thenumber of seconds since a particular epoch. On POSIX-compliant systems,it is the number of seconds since1970-01-01 00:00:00 UTC, not counting leap seconds.45All known POSIX-compliant systems support timestamps from 0 through2^31 - 1, which is sufficient to represent times through2038-01-19 03:14:07 UTC. Many systems support a wider range of timestamps,including negative timestamps that represent times before theepoch.
In order to make it easier to process such log files and to produceuseful reports,gawk provides the following functions forworking with timestamps. They aregawk extensions; they arenot specified in the POSIX standard, nor are they in any other knownversion ofawk.46Optional parameters are enclosed in square brackets ([ ]):
mktime(
datespec
)
systime()
. It is similar to the function of thesame name in ISO C. The argument, datespec, is a string of the form
"
YYYY
MM
DD
HH
MM
SS
[
DST
]"
. The string consists of six or seven numbers representing, respectively,the full year including century, the month from 1 to 12, the day of the monthfrom 1 to 31, the hour of the day from 0 to 23, the minute from 0 to59, the second from 0 to 60,
47and an optional daylight-savings flag.
The values of these numbers need not be within the ranges specified;for example, an hour of −1 means 1 hour before midnight. The origin-zero Gregorian calendar is assumed, with year 0 precedingyear 1 and year −1 preceding year 0. The time is assumed to be in the local timezone. If the daylight-savings flag is positive, the time is assumed to bedaylight savings time; if zero, the time is assumed to be standardtime; and if negative (the default),mktime()
attempts to determinewhether daylight savings time is in effect for the specified time.
If datespec does not contain enough elements or if the resulting timeis out of range,mktime()
returns −1.
strftime(
[ format
[
,
timestamp
[
,
utc-flag
]]]
)
systime()
function. If no timestamp argument is supplied,
gawk uses the current time of day as the timestamp. If no format argument is supplied,
strftime()
usesthe value of
PROCINFO["strftime"]
as the format string(see Built-in Variables). The default string value is
"%a %b %e %H:%M:%S %Z %Y"
. This format string producesoutput that is equivalent to that of the
date utility. You can assign a new value to
PROCINFO["strftime"]
tochange the default format.
systime()
The systime()
function allows you to compare a timestamp from alog file with the current time of day. In particular, it is easy todetermine how long ago a particular record was logged. It also allowsyou to produce log records using the “seconds since the epoch” format.
Themktime()
function allows you to convert a textual representationof a date and time into a timestamp. This makes it easy to do before/aftercomparisons of dates and times, particularly when dealing with date andtime data coming from an external source, such as a log file.
The strftime()
function allows you to easily turn a timestampinto human-readable information. It is similar in nature to thesprintf()
function(see String Functions),in that it copies nonformat specification characters verbatim to thereturned string, while substituting date and time values for formatspecifications in theformat string.
strftime()
is guaranteed by the 1999 ISO Cstandard48to support the following date format specifications:
%a
%A
%b
%B
%c
"C"
locale.)
%C
%d
%D
%e
%F
%g
%G
%h
%H
%I
%j
%m
%M
%n
%p
%r
"C"
locale.)
%R
%S
%t
%T
%u
%U
%V
%w
%W
%x
"C"
locale.)
%X
"C"
locale.)
%y
%Y
%z
%Z
%Ec %EC %Ex %EX %Ey %EY %Od %Oe %OH
%OI %Om %OM %OS %Ou %OU %OV %Ow %OW %Oy
%%
If a conversion specifier is not one of the above, the behavior isundefined.50
Informally, a locale is the geographic place in which a programis meant to run. For example, a common way to abbreviate the dateSeptember 4, 2012 in the United States is “9/4/12.”In many countries in Europe, however, it is abbreviated “4.9.12.”Thus, the ‘%x’ specification in a "US"
locale might produce‘9/4/12’, while in a"EUROPE"
locale, it might produce‘4.9.12’. The ISO C standard defines a default"C"
locale, which is an environment that is typical of what many C programmersare used to.
For systems that are not yet fully standards-compliant,gawk supplies a copy ofstrftime()
from the GNU C Library. It supports all of the just-listed format specifications. If that version isused to compilegawk (see Installation),then the following additional format specifications are available:
%k
%l
%s
Additionally, the alternate representations are recognized but theirnormal representations are used.
The following example is anawk implementation of the POSIXdate utility. Normally, thedate utility prints thecurrent date and time of day in a well-known format. However, if youprovide an argument to it that begins with a ‘+’,datecopies nonformat specifier characters to the standard output andinterprets the current time according to the format specifiers inthe string. For example:
$ date '+Today is %A, %B %d, %Y.' -| Today is Wednesday, March 30, 2011.
Here is the gawk version of the date utility. It has a shell “wrapper” to handle the-u option,which requires that date run as if the time zoneis set to UTC:
#! /bin/sh # # date --- approximate the POSIX 'date' command case $1 in -u) TZ=UTC0 # use UTC export TZ shift ;; esac gawk 'BEGIN { format = "%a %b %e %H:%M:%S %Z %Y" exitval = 0 if (ARGC > 2) exitval = 1 else if (ARGC == 2) { format = ARGV[1] if (format ~ /^\+/) format = substr(format, 2) # remove leading + } print strftime(format) exit exitval }' "$@"
I can explain it for you, but I can't understand it for you.
Anonymous
Many languages provide the ability to perform bitwise operationson two integer numbers. In other words, the operation is performed oneach successive pair of bits in the operands. Three common operations are bitwise AND, OR, and XOR. The operations are described in table-bitwise-ops.
Bit Operator | AND | OR | XOR |—+—+—+—+—+— Operands | 0 | 1 | 0 | 1 | 0 | 1 ————–+—+—+—+—+—+— 0 | 0 0 | 0 1 | 0 1 1 | 0 1 | 1 1 | 1 0
Table 9.4: Bitwise Operations
As you can see, the result of an AND operation is 1 only whenbothbits are 1. The result of an OR operation is 1 if either bit is 1. The result of an XOR operation is 1 if either bit is 1,but not both. The next operation is thecomplement; the complement of 1 is 0 andthe complement of 0 is 1. Thus, this operation “flips” all the bitsof a given value.
Finally, two other common operations are to shift the bits left or right. For example, if you have a bit string ‘10111001’ and you shift itright by three bits, you end up with ‘00010111’.51If you start overagain with ‘10111001’ and shift it left by three bits, you end upwith ‘11001000’.gawk provides built-in functions that implement thebitwise operations just described. They are:
and(
v1
,
v2
)
compl(
val
)
lshift(
val
,
count
)
or(
v1
,
v2
)
rshift(
val
,
count
)
xor(
v1
,
v2
)
For all of these functions, first the double precision floating-point value isconverted to the widest C unsigned integer type, then the bitwise operation isperformed. If the result cannot be represented exactly as a Cdouble
,leading nonzero bits are removed one by one until it can be representedexactly. The result is then converted back into a Cdouble
. (Ifyou don't understand this paragraph, don't worry about it.)
Here is a user-defined function (see User-defined)that illustrates the use of these functions:
# bits2str --- turn a byte into readable 1's and 0's function bits2str(bits, data, mask) { if (bits == 0) return "0" mask = 1 for (; bits != 0; bits = rshift(bits, 1)) data = (and(bits, mask) ? "1" : "0") data while ((length(data) % 8) != 0) data = "0" data return data } BEGIN { printf "123 = %s\n", bits2str(123) printf "0123 = %s\n", bits2str(0123) printf "0x99 = %s\n", bits2str(0x99) comp = compl(0x99) printf "compl(0x99) = %#x = %s\n", comp, bits2str(comp) shift = lshift(0x99, 2) printf "lshift(0x99, 2) = %#x = %s\n", shift, bits2str(shift) shift = rshift(0x99, 2) printf "rshift(0x99, 2) = %#x = %s\n", shift, bits2str(shift) }
This program produces the following output when run:
$ gawk -f testbits.awk -| 123 = 01111011 -| 0123 = 01010011 -| 0x99 = 10011001 -| compl(0x99) = 0xffffff66 = 11111111111111111111111101100110 -| lshift(0x99, 2) = 0x264 = 0000001001100100 -| rshift(0x99, 2) = 0x26 = 00100110
Thebits2str()
function turns a binary number into a string. The number 1
represents a binary value where the rightmost bitis set to 1. Using this mask,the function repeatedly checks the rightmost bit. ANDing the mask with the value indicates whether therightmost bit is 1 or not. If so, a"1"
is concatenated onto the frontof the string. Otherwise, a "0"
is added. The value is then shifted right by one bit and the loop continuesuntil there are no more 1 bits.
If the initial value is zero it returns a simple "0"
. Otherwise, at the end, it pads the value with zeros to represent multiplesof 8-bit quantities. This is typical in modern computers.
The main code in the BEGIN
rule shows the difference between thedecimal and octal values for the same numbers(seeNondecimal-numbers),and then demonstrates theresults of thecompl()
, lshift()
, and rshift()
functions.
gawk provides a single function that lets you distinguishan array from a scalar variable. This is necessary for writing codethat traverses every element of a true multidimensional array(seeArrays of Arrays).
isarray(
x
)
gawk provides facilities for internationalizing awk programs. These include the functions described in the following list. The descriptions here are purposely brief. SeeInternationalization,for the full story. Optional parameters are enclosed in square brackets ([ ]):
bindtextdomain(
directory
[
,
domain
]
)
The default domain is the value of TEXTDOMAIN
. If directory is the null string (""
), thenbindtextdomain()
returns the current binding for thegivendomain.
dcgettext(
string
[
,
domain
[
,
category
]]
)
TEXTDOMAIN
. The default value for category is
"LC_MESSAGES"
.
dcngettext(
string1
,
string2
,
number
[
,
domain
[
,
category
]]
)
TEXTDOMAIN
. The default value for category is
"LC_MESSAGES"
.
Complicatedawk programs can often be simplified by definingyour own functions. User-defined functions can be called just likebuilt-in ones (seeFunction Calls), but it is up to you to definethem, i.e., to tellawk what they should do.
Definitions of functions can appear anywhere between the rules of anawk program. Thus, the general form of anawk program isextended to include sequences of rulesand user-defined functiondefinitions. There is no need to put the definition of a functionbefore all uses of the function. This is becauseawk reads theentire program before starting to execute any of it.
The definition of a function named name looks like this:
function name([parameter-list]) { body-of-function }
Here,name is the name of the function to define. A valid functionname is like a valid variable name: a sequence of letters, digits, andunderscores that doesn't start with a digit. Within a singleawk program, any particular name can only beused as a variable, array, or function.
parameter-list is an optional list of the function's arguments and localvariable names, separated by commas. When the function is called,the argument names are used to hold the argument values given inthe call. The local variables are initialized to the empty string. A function cannot have two parameters with the same name, nor may ithave a parameter with the same name as the function itself.
In addition, according to the POSIX standard, function parameters cannot have the samename as one of the special built-in variables(seeBuilt-in Variables. Not all versions of awkenforce this restriction.
The body-of-function consists of awk statements. It is themost important part of the definition, because it says what the functionshould actuallydo. The argument names exist to give the body away to talk about the arguments; local variables exist to give the bodyplaces to keep temporary values.
Argument names are not distinguished syntactically from local variablenames. Instead, the number of arguments supplied when the function iscalled determines how many argument variables there are. Thus, if threeargument values are given, the first three names in parameter-listare arguments and the rest are local variables.
It follows that if the number of arguments is not the same in all callsto the function, some of the names inparameter-list may bearguments on some occasions and local variables on others. Anotherway to think of this is that omitted arguments default to thenull string.
Usually when you write a function, you know how many names you intend touse for arguments and how many you intend to use as local variables. It isconventional to place some extra space between the arguments andthe local variables, in order to document how your function is supposed to be used.
During execution of the function body, the arguments and local variablevalues hide, orshadow, any variables of the same names used in therest of the program. The shadowed variables are not accessible in thefunction definition, because there is no way to name them while theirnames have been taken away for the local variables. All other variablesused in the awk program can be referenced or set normally in thefunction's body.
The arguments and local variables last only as long as the function bodyis executing. Once the body finishes, you can once again access thevariables that were shadowed while the function was running.
The function body can contain expressions that call functions. Theycan even call this function, either directly or by way of anotherfunction. When this happens, we say the function is recursive. The act of a function calling itself is calledrecursion.
All the built-in functions return a value to their caller. User-defined functions can do also, using thereturn
statement,which is described in detail in Return Statement. Many of the subsequent examples in this section usethe return
statement.
In many awk implementations, including gawk,the keyword function
may beabbreviatedfunc
. (c.e.) However, POSIX only specifies the use ofthe keyword function
. This actually has some practical implications. If gawk is in POSIX-compatibility mode(seeOptions), then the followingstatement does not define a function:
func foo() { a = sqrt($1) ; print a }
Instead it defines a rule that, for each record, concatenates the valueof the variable ‘func’ with the return value of the function ‘foo’. If the resulting string is non-null, the action is executed. This is probably not what is desired. (awk accepts this input assyntactically valid, because functions may be used before they are definedinawk programs.52)
To ensure that yourawk programs are portable, always use thekeywordfunction
when defining a function.
Here is an example of a user-defined function, called myprint()
, thattakes a number and prints it in a specific format:
function myprint(num) { printf "%6.3g\n", num }
To illustrate, here is an awk rule that uses ourmyprint
function:
$3 > 0 { myprint($3) }
This program prints, in our special format, all the third fields thatcontain a positive number in our input. Therefore, when given the following input:
1.2 3.4 5.6 7.8 9.10 11.12 -13.14 15.16 17.18 19.20 21.22 23.24
this program, using our function to format the results, prints:
5.6 21.2
This function deletes all the elements in an array:
function delarray(a, i) { for (i in a) delete a[i] }
When working with arrays, it is often necessary to delete all the elementsin an array and start over with a new list of elements(seeDelete). Instead of havingto repeat this loop everywhere that you need to clear outan array, your program can just calldelarray
. (This guarantees portability. The use of ‘deletearray’ to deletethe contents of an entire array is a nonstandard extension.)
The following is an example of a recursive function. It takes a stringas an input parameter and returns the string in backwards order. Recursive functions must always have a test that stops the recursion. In this case, the recursion terminates when the starting positionis zero, i.e., when there are no more characters left in the string.
function rev(str, start) { if (start == 0) return "" return (substr(str, start, 1) rev(str, start - 1)) }
If this function is in a file named rev.awk, it can be testedthis way:
$ echo "Don't Panic!" | > gawk --source '{ print rev($0, length($0)) }' -f rev.awk -| !cinaP t'noD
The C ctime()
function takes a timestamp and returns it in a string,formatted in a well-known fashion. The following example uses the built-instrftime()
function(see Time Functions)to create anawk version of ctime()
:
# ctime.awk # # awk version of C ctime(3) function function ctime(ts, format) { format = "%a %b %e %H:%M:%S %Z %Y" if (ts == 0) ts = systime() # use current time as default return strftime(format, ts) }
This section describes how to call a user-defined function.
Calling a function means causing the function to run and do its job. A function call is an expression and its value is the value returned bythe function.
A function call consists of the function name followed by the argumentsin parentheses.awk expressions are what you write in thecall for the arguments. Each time the call is executed, theseexpressions are evaluated, and the values become the actual arguments. Forexample, here is a call tofoo()
with three arguments (the firstbeing a string concatenation):
foo(x y, "lose", 4 * z)
CAUTION: Whitespace characters (spaces and TABs) are not allowedbetween the function name and the open-parenthesis of the argument list. If you write whitespace by mistake, awk might think that you meanto concatenate a variable with an expression in parentheses. However, itnotices that you used a function name and not a variable name, and reportsan error.
There is no way to make a variable local to a{ ... }
block inawk, but you can make a variable local to a function. It isgood practice to do so whenever a variable is needed only in thatfunction.
To make a variable local to a function, simply declare the variable asan argument after the actual function arguments(seeDefinition Syntax). Look at the following example where variablei
is a global variable used by both functionsfoo()
andbar()
:
function bar() { for (i = 0; i < 3; i++) print "bar's i=" i } function foo(j) { i = j + 1 print "foo's i=" i bar() print "foo's i=" i } BEGIN { i = 10 print "top's i=" i foo(0) print "top's i=" i }
Running this script produces the following, because the i
infunctionsfoo()
and bar()
and at the top level refer to the samevariable instance:
top's i=10 foo's i=1 bar's i=0 bar's i=1 bar's i=2 foo's i=3 top's i=3
If you want i
to be local to both foo()
and bar()
do asfollows (the extra-space beforei
is a coding convention toindicate that i
is a local variable, not an argument):
function bar( i) { for (i = 0; i < 3; i++) print "bar's i=" i } function foo(j, i) { i = j + 1 print "foo's i=" i bar() print "foo's i=" i } BEGIN { i = 10 print "top's i=" i foo(0) print "top's i=" i }
Running the corrected script produces the following:
top's i=10 foo's i=1 bar's i=0 bar's i=1 bar's i=2 foo's i=1 top's i=10
In awk, when you declare a function, there is no way todeclare explicitly whether the arguments are passedby value orby reference.
Instead the passing convention is determined at runtime whenthe function is called according to the following rule:
Passing an argument by value means that when a function is called, itis given acopy of the value of this argument. The caller may use a variable as the expression for the argument, butthe called function does not know this—it only knows what value theargument had. For example, if you write the following code:
foo = "bar" z = myfunc(foo)
then you should not think of the argument to myfunc()
as being“the variablefoo
.” Instead, think of the argument as thestring value "bar"
. If the functionmyfunc()
alters the values of its local variables,this has no effect on any other variables. Thus, ifmyfunc()
does this:
function myfunc(str) { print str str = "zzz" print str }
to change its first argument variable str
, it doesnotchange the value of foo
in the caller. The role of foo
incalling myfunc()
ended when its value ("bar"
) was computed. Ifstr
also exists outside of myfunc()
, the function bodycannot alter this outer value, because it is shadowed during theexecution ofmyfunc()
and cannot be seen or changed from there.
However, when arrays are the parameters to functions, they arenotcopied. Instead, the array itself is made available for direct manipulationby the function. This is usually termedcall by reference. Changes made to an array parameter inside the body of a functionarevisible outside that function.
NOTE: Changing an array parameter inside a functioncan be very dangerous if you do not watch what you are doing. For example:function changeit(array, ind, nvalue) { array[ind] = nvalue } BEGIN { a[1] = 1; a[2] = 2; a[3] = 3 changeit(a, 2, "two") printf "a[1] = %s, a[2] = %s, a[3] = %s\n", a[1], a[2], a[3] }prints ‘a[1] = 1, a[2] = two, a[3] = 3’, because
changeit
stores"two"
in the second element ofa
.
Someawk implementations allow you to call a function thathas not been defined. They only report a problem at runtime when theprogram actually tries to call the function. For example:
BEGIN { if (0) foo() else bar() } function bar() { ... } # note that `foo' is not defined
Because the ‘if’ statement will never be true, it is not really aproblem thatfoo()
has not been defined. Usually, though, it is aproblem if a program calls an undefined function.
If --lint is specified(seeOptions),gawk reports calls to undefined functions.
Someawk implementations generate a runtimeerror if you use thenext
statement(see Next Statement)inside a user-defined function.gawk does not have this limitation.
return
StatementAs seen in several earlier examples,the body of a user-defined function can contain areturn
statement. This statement returns control to the calling part of theawk program. Itcan also be used to return a value for use in the rest of theawkprogram. It looks like this:
return [expression]
The expression part is optional. Due most likely to an oversight, POSIX does not define what the returnvalue is if you omit theexpression. Technically speaking, thismake the returned value undefined, and therefore, unpredictable. In practice, though, all versions ofawk simply return thenull string, which acts like zero if used in a numeric context.
A return
statement with no value expression is assumed at the end ofevery function definition. So if control reaches the end of the functionbody, then technically, the function returns an unpredictable value. In practice, it returns the empty string. awkdoes not warn you if you use the return value of such a function.
Sometimes, you want to write a function for what it does, not forwhat it returns. Such a function corresponds to avoid
functionin C, C++ or Java, or to a procedure
in Ada. Thus, it may be appropriate to notreturn any value; simply bear in mind that you should not be using thereturn value of such a function.
The following is an example of a user-defined function that returns a valuefor the largest number among the elements of an array:
function maxelt(vec, i, ret) { for (i in vec) { if (ret == "" || vec[i] > ret) ret = vec[i] } return ret }
You callmaxelt()
with one argument, which is an array name. The localvariablesi
and ret
are not intended to be arguments;while there is nothing to stop you from passing more than one argumenttomaxelt()
, the results would be strange. The extra space beforei
in the function parameter list indicates thati
andret
are local variables. You should follow this convention when defining functions.
The following program uses the maxelt()
function. It loads anarray, callsmaxelt()
, and then reports the maximum number in thatarray:
function maxelt(vec, i, ret) { for (i in vec) { if (ret == "" || vec[i] > ret) ret = vec[i] } return ret } # Load all fields of each record into nums. { for(i = 1; i <= NF; i++) nums[NR, i] = $i } END { print maxelt(nums) }
Given the following input:
1 5 23 8 16 44 3 5 2 8 26 256 291 1396 2962 100 -6 467 998 1101 99385 11 0 225
the program reports (predictably) that 99,385 is the largest valuein the array.
awk is a very fluid language. It is possible thatawk can't tell if an identifierrepresents a scalar variable or an array until runtime. Here is an annotated sample program:
function foo(a) { a[1] = 1 # parameter is an array } BEGIN { b = 1 foo(b) # invalid: fatal type mismatch foo(x) # x uninitialized, becomes an array dynamically x = 1 # now not allowed, runtime error }
Usually, such things aren't a big issue, but it's worthbeing aware of them.
This section describes a gawk-specific extension.
Often, you may wish to defer the choice of function to call until runtime. For example, you may have different kinds of records, each of whichshould be processed differently.
Normally, you would have to use a series of if
-else
statements to decide which function to call. By usingindirectfunction calls, you can specify the name of the function to call as astring variable, and then call the function. Let's look at an example.
Suppose you have a file with your test scores for the classes youare taking. The first field is the class name. The following fieldsare the functions to call to process the data, up to a “marker”field ‘data:’. Following the marker, to the end of the record,are the various numeric test scores.
Here is the initial file; you wish to get the sum and the average ofyour test scores:
Biology_101 sum average data: 87.0 92.4 78.5 94.9 Chemistry_305 sum average data: 75.2 98.3 94.7 88.2 English_401 sum average data: 100.0 95.6 87.1 93.4
To process the data, you might write initially:
{ class = $1 for (i = 2; $i != "data:"; i++) { if ($i == "sum") sum() # processes the whole record else if ($i == "average") average() ... # and so on } }
This style of programming works, but can be awkward. With indirectfunction calls, you tell gawk to use thevalue of avariable as the name of the function to call.
The syntax is similar to that of a regular function call: an identifierimmediately followed by a left parenthesis, any arguments, and thena closing right parenthesis, with the addition of a leading ‘@’character:
the_func = "sum" result = @the_func() # calls the `sum' function
Here is a full program that processes the previously shown data,using indirect function calls.
# indirectcall.awk --- Demonstrate indirect function calls # average --- return the average of the values in fields $first - $last function average(first, last, sum, i) { sum = 0; for (i = first; i <= last; i++) sum += $i return sum / (last - first + 1) } # sum --- return the sum of the values in fields $first - $last function sum(first, last, ret, i) { ret = 0; for (i = first; i <= last; i++) ret += $i return ret }
These two functions expect to work on fields; thus the parametersfirst
andlast
indicate where in the fields to start and end. Otherwise they perform the expected computations and are not unusual.
# For each record, print the class name and the requested statistics { class_name = $1 gsub(/_/, " ", class_name) # Replace _ with spaces # find start for (i = 1; i <= NF; i++) { if ($i == "data:") { start = i + 1 break } } printf("%s:\n", class_name) for (i = 2; $i != "data:"; i++) { the_function = $i printf("\t%s: <%s>\n", $i, @the_function(start, NF) "") } print "" }
This is the main processing for each record. It prints the class name (withunderscores replaced with spaces). It then finds the start of the actual data,saving it instart
. The last part of the code loops through each function name (from$2
up tothe marker, ‘data:’), calling the function named by the field. The indirectfunction call itself occurs as a parameter in the call toprintf
. (The printf
format string uses ‘%s’ as the format specifier so that wecan use functions that return strings, as well as numbers. Note that the resultfrom the indirect call is concatenated with the empty string, in order to forceit to be a string value.)
Here is the result of running the program:
$ gawk -f indirectcall.awk class_data1 -| Biology 101: -| sum: <352.8> -| average: <88.2> -| -| Chemistry 305: -| sum: <356.4> -| average: <89.1> -| -| English 401: -| sum: <376.1> -| average: <94.025>
The ability to use indirect function calls is more powerful than you maythink at first. The C and C++ languages provide “function pointers,” whichare a mechanism for calling a function chosen at runtime. One of the mostwell-known uses of this ability is the C qsort()
function, which sortsan array using the famous “quick sort” algorithm(seethe Wikipedia articlefor more information). To use this function, you supply a pointer to a comparisonfunction. This mechanism allows you to sort arbitrary data in an arbitraryfashion.
We can do something similar using gawk, like this:
# quicksort.awk --- Quicksort algorithm, with user-supplied # comparison function # quicksort --- C.A.R. Hoare's quick sort algorithm. See Wikipedia # or almost any algorithms or computer science text function quicksort(data, left, right, less_than, i, last) { if (left >= right) # do nothing if array contains fewer return # than two elements quicksort_swap(data, left, int((left + right) / 2)) last = left for (i = left + 1; i <= right; i++) if (@less_than(data[i], data[left])) quicksort_swap(data, ++last, i) quicksort_swap(data, left, last) quicksort(data, left, last - 1, less_than) quicksort(data, last + 1, right, less_than) } # quicksort_swap --- helper function for quicksort, should really be inline function quicksort_swap(data, i, j, temp) { temp = data[i] data[i] = data[j] data[j] = temp }
The quicksort()
function receives the data
array, the starting and endingindices to sort (left
andright
), and the name of a function thatperforms a “less than” comparison. It then implements the quick sort algorithm.
To make use of the sorting function, we return to our previous example. Thefirst thing to do is write some comparison functions:
# num_lt --- do a numeric less than comparison function num_lt(left, right) { return ((left + 0) < (right + 0)) } # num_ge --- do a numeric greater than or equal to comparison function num_ge(left, right) { return ((left + 0) >= (right + 0)) }
The num_ge()
function is needed to perform a descending sort; when usedto perform a “less than” test, it actually does the opposite (greater thanor equal to), which yields data sorted in descending order.
Next comes a sorting function. It is parameterized with the starting andending field numbers and the comparison function. It builds an array withthe data and callsquicksort
appropriately, and then formats theresults as a single string:
# do_sort --- sort the data according to `compare' # and return it as a string function do_sort(first, last, compare, data, i, retval) { delete data for (i = 1; first <= last; first++) { data[i] = $first i++ } quicksort(data, 1, i-1, compare) retval = data[1] for (i = 2; i in data; i++) retval = retval " " data[i] return retval }
Finally, the two sorting functions call do_sort()
, passing in thenames of the two comparison functions:
# sort --- sort the data in ascending order and return it as a string function sort(first, last) { return do_sort(first, last, "num_lt") } # rsort --- sort the data in descending order and return it as a string function rsort(first, last) { return do_sort(first, last, "num_ge") }
Here is an extended version of the data file:
Biology_101 sum average sort rsort data: 87.0 92.4 78.5 94.9 Chemistry_305 sum average sort rsort data: 75.2 98.3 94.7 88.2 English_401 sum average sort rsort data: 100.0 95.6 87.1 93.4
Finally, here are the results when the enhanced program is run:
$ gawk -f quicksort.awk -f indirectcall.awk class_data2 -| Biology 101: -| sum: <352.8> -| average: <88.2> -| sort: <78.5 87.0 92.4 94.9> -| rsort: <94.9 92.4 87.0 78.5> -| -| Chemistry 305: -| sum: <356.4> -| average: <89.1> -| sort: <75.2 88.2 94.7 98.3> -| rsort: <98.3 94.7 88.2 75.2> -| -| English 401: -| sum: <376.1> -| average: <94.025> -| sort: <87.1 93.4 95.6 100.0> -| rsort: <100.0 95.6 93.4 87.1>
Remember that you must supply a leading ‘@’ in front of an indirect function call.
Unfortunately, indirect function calls cannot be used with the built-in functions. However,you can generally write “wrapper” functions which call the built-in ones, and those canbe called indirectly. (Other than, perhaps, the mathematical functions, there is not a lotof reason to try to call the built-in functions indirectly.)
gawk does its best to make indirect function calls efficient. For example, in the following case:
for (i = 1; i <= n; i++) @the_func()
gawk
will look up the actual function to call only once.
Once upon a time, computer makerswrote software that worked only in English. Eventually, hardware and software vendors noticed that if theirsystems worked in the native languages of non-English-speakingcountries, they were able to sell more systems. As a result, internationalization and localizationof programs and software systems became a common practice.
For many years, the ability to provide internationalizationwas largely restricted to programs written in C and C++. This chapter describes the underlying librarygawkuses for internationalization, as well as howgawk makes internationalizationfeatures available at theawk program level. Having internationalization available at theawk levelgives software developers additional flexibility—they are nolonger forced to write in C or C++ when internationalization isa requirement.
Internationalization means writing (or modifying) a program once,in such a way that it can use multiple languages without requiringfurther source-code changes.Localization means providing the data necessary for aninternationalized program to work in a particular language. Most typically, these terms refer to features such as the languageused for printing error messages, the language used to readresponses, and information related to how numerical andmonetary values are printed and read.
gettext
The facilities in GNUgettext
focus on messages; strings printedby a program, either directly or via formatting withprintf
orsprintf()
.53
When using GNUgettext
, each application has its owntext domain. This is a unique name, such as ‘kpilot’ or ‘gawk’,that identifies the application. A complete application may have multiple components—programs writtenin C or C++, as well as scripts written insh or awk. All of the components use the same text domain.
To make the discussion concrete, assume we're writing an applicationnamed guide. Internationalization consists of thefollowing steps, in this order:
"`-F': option required"
is a good candidate for translation. A table with strings of option names is not (e.g.,gawk's--profile option should remain the same, no matter what the locallanguage). "guide"
) to thegettext
library,by calling the textdomain()
function. .po
)and translations are created and shipped with the application. For example, there might be afr.po for a French translation. gettext
to use.mo files in a different directory than the standardone by using thebindtextdomain()
function. gettext()
. The returned string is the translated stringif available, or the original string if not.In C (or C++), the string marking and dynamic translation lookupare accomplished by wrapping each string in a call togettext()
:
printf("%s", gettext("Don't Panic!\n"));
The tools that extract messages from source code pull out allstrings enclosed in calls togettext()
.
The GNUgettext
developers, recognizing that typing‘gettext(...)’ over and over again is both painful and ugly to lookat, use the macro ‘_’ (an underscore) to make things easier:
/* In the standard header file: */ #define _(str) gettext(str) /* In the program text: */ printf("%s", _("Don't Panic!\n"));
This reduces the typing overhead to just three extra characters per stringand is considerably easier to read as well.
There are locale categoriesfor different types of locale-related information. The defined locale categories thatgettext
knows about are:
LC_MESSAGES
gettext
operations, but it is possible to supply a different one explicitly,if necessary. (It is almost never necessary to supply a different category.)
LC_COLLATE
LC_CTYPE
/[[:alnum:]]/
(see Regexp Operators).
LC_MONETARY
LC_NUMERIC
LC_RESPONSE
LC_TIME
LC_ALL
gettext
.)
gawk provides the following variables and functions forinternationalization:
TEXTDOMAIN
gettext
, the defaultvalue is
"messages"
.
_"your message here"
dcgettext(
string
[
,
domain
[
,
category
]]
)
TEXTDOMAIN
. The default value for category is
"LC_MESSAGES"
.
If you supply a value for category, it must be a string equal toone of the known locale categories described inthe previous section. You must also supply a text domain. UseTEXTDOMAIN
ifyou want to use the current domain.
CAUTION: The order of arguments to the
awk versionof the
dcgettext()
function is purposely different from the order forthe C version. The
awk version's order waschosen to be simple and to allow for reasonable
awk-styledefault arguments.
dcngettext(
string1
,
string2
,
number
[
,
domain
[
,
category
]]
)
TEXTDOMAIN
. The default value for category is
"LC_MESSAGES"
.
The same remarks about argument order as for the dcgettext()
function apply.
bindtextdomain(
directory
[
,
domain
]
)
gettext
looks for
.mo files, in case theywill not or cannot be placed in the standard locations(e.g., during testing). Return the directory in which domain is “bound.”
The default domain is the value of TEXTDOMAIN
. If directory is the null string (""
), thenbindtextdomain()
returns the current binding for thegivendomain.
To use these facilities in your awk program, follow the stepsoutlined inthe previous section,like so:
TEXTDOMAIN
to the text domain ofyour program. This is best done in aBEGIN
rule(see BEGIN/END),or it can also be done via the-v command-lineoption (see Options): BEGIN { TEXTDOMAIN = "guide" ... }
print _"hello, world" x = _"you goofed" printf(_"Number of users is %d\n", nusers)
dcgettext()
built-in function: message = nusers " users logged in" message = dcgettext(message, "adminprog") print message
Here, the call to dcgettext()
supplies a differenttext domain ("adminprog"
) in which to find themessage, but it uses the default"LC_MESSAGES"
category.
bindtextdomain()
built-in function: BEGIN { TEXTDOMAIN = "guide" # our text domain if (Testing) { # where to find our files bindtextdomain("testdir") # joe is in charge of adminprog bindtextdomain("../joe/testdir", "adminprog") } ... }
See I18N Example,for an example program showing the steps to createand use translations fromawk.
Once a program's translatable strings have been marked, they mustbe extracted to create the initial .po file. As part of translation, it is often helpful to rearrange the orderin which arguments toprintf
are output.
gawk's --gen-pot command-line option extractsthe messages and is discussed next. After that,printf
's ability torearrange the order for printf
arguments at runtimeis covered.
Once your awk program is working, and all the strings havebeen marked and you've set (and perhaps bound) the text domain,it is time to produce translations. First, use the--gen-pot command-line option to createthe initial.pot file:
$ gawk --gen-pot -f guide.awk > guide.pot
When run with --gen-pot, gawk does not execute yourprogram. Instead, it parses it as usual and prints all marked stringsto standard output in the format of a GNUgettext
Portable Objectfile. Also included in the output are any constant strings thatappear as the first argument todcgettext()
or as the first andsecond argument to dcngettext()
.55SeeI18N Example,for the full list of steps to go through to create and testtranslations forguide.
printf
ArgumentsFormat strings forprintf
and sprintf()
(see Printf)present a special problem for translation. Consider the following:56
printf(_"String `%s' has %d characters\n", string, length(string)))
A possible German translation for this might be:
"%d Zeichen lang ist die Zeichenkette `%s'\n"
The problem should be obvious: the order of the formatspecifications is different from the original! Even thoughgettext()
can return the translated stringat runtime,it cannot change the argument order in the call toprintf
.
To solve this problem, printf
format specifiers may havean additional optional element, which we call apositional specifier. For example:
"%2$d Zeichen lang ist die Zeichenkette `%1$s'\n"
Here, the positional specifier consists of an integer count, which indicates whichargument to use, and a ‘$’. Counts are one-based, and theformat string itself isnot included. Thus, in the followingexample, ‘string’ is the first argument and ‘length(string)’ is the second:
$ gawk 'BEGIN { > string = "Dont Panic" > printf _"%2$d characters live in \"%1$s\"\n", > string, length(string) > }' -| 10 characters live in "Dont Panic"
If present, positional specifiers come first in the format specification,before the flags, the field width, and/or the precision.
Positional specifiers can be used with the dynamic field width andprecision capability:
$ gawk 'BEGIN { > printf("%*.*s\n", 10, 20, "hello") > printf("%3$*2$.*1$s\n", 20, 10, "hello") > }' -| hello -| hello
NOTE: When using ‘ *’ with a positional specifier, the ‘ *’comes first, then the integer position, and then the ‘ $’. This is somewhat counterintuitive.
gawk does not allow you to mix regular format specifiersand those with positional specifiers in the same string:
$ gawk 'BEGIN { printf _"%d %3$s\n", 1, 2, "hi" }' error--> gawk: cmd. line:1: fatal: must use `count$' on all formats or none
NOTE: There are some pathological cases that gawk may fail todiagnose. In such cases, the output may not be what you expect. It's still a bad idea to try mixing them, even if gawkdoesn't detect it.
Although positional specifiers can be used directly in awk programs,their primary purpose is to help in producing correct translations offormat strings into languages different from the one in which the programis first written.
gawk's internationalization features were purposely chosen tohave as little impact as possible on the portability of awkprograms that use them to other versions ofawk. Consider this program:
BEGIN { TEXTDOMAIN = "guide" if (Test_Guide) # set with -v bindtextdomain("/test/guide/messages") print _"don't panic!" }
As written, it won't work on other versions of awk. However, it is actually almost portable, requiring very littlechange:
TEXTDOMAIN
won't have any effect,since TEXTDOMAIN
is not special in otherawk implementations. _
with the stringfollowing it.57 Typically, the variable_
hasthe null string (""
) as its value, leaving the original string constant asthe result.dcgettext()
, dcngettext()
andbindtextdomain()
, the awk program can be made to run, butall the messages are output in the original language. For example: function bindtextdomain(dir, domain) { return dir } function dcgettext(string, domain, category) { return string } function dcngettext(string1, string2, number, domain, category) { return (number == 1 ? string1 : string2) }
printf
orsprintf()
isnot portable. To support gettext()
at the C level, many systems' C versions ofsprintf()
do support positional specifiers. But it works only ifenough arguments are supplied in the function call. Many versions ofawk pass printf
formats and arguments unchanged to theunderlying C library version ofsprintf()
, but only one format andargument at a time. What happens if a positional specification isused is anybody's guess. However, since the positional specifications are primarily for use intranslated format strings, and since non-GNUawks neverretrieve the translated string, this should not be a problem in practice. Now let's look at a step-by-step example of how to internationalize andlocalize a simpleawk program, using guide.awk as ouroriginal source:
BEGIN { TEXTDOMAIN = "guide" bindtextdomain(".") # for testing print _"Don't Panic" print _"The Answer Is", 42 print "Pardon me, Zaphod who?" }
Run ‘gawk --gen-pot’ to create the.pot file:
$ gawk --gen-pot -f guide.awk > guide.pot
This produces:
#: guide.awk:4 msgid "Don't Panic" msgstr "" #: guide.awk:5 msgid "The Answer Is" msgstr ""
This original portable object template file is saved and reused for each languageinto which the application is translated. Themsgid
is the original string and the msgstr
is the translation.
NOTE: Strings not marked with a leading underscore do notappear in the guide.pot file.
Next, the messages must be translated. Here is a translation to a hypothetical dialect of English,called “Mellow”:58
$ cp guide.pot guide-mellow.po Add translations to guide-mellow.po ...
Following are the translations:
#: guide.awk:4 msgid "Don't Panic" msgstr "Hey man, relax!" #: guide.awk:5 msgid "The Answer Is" msgstr "Like, the scoop is"
The next step is to make the directory to hold the binary message objectfile and then to create theguide.mo file. The directory layout shown here is standard for GNUgettext
onGNU/Linux systems. Other versions of gettext
may use a differentlayout:
$ mkdir en_US en_US/LC_MESSAGES
Themsgfmt utility does the conversion from human-readable.po file to machine-readable.mo file. By default, msgfmt creates a file namedmessages. This file must be renamed and placed in the proper directory so thatgawk can find it:
$ msgfmt guide-mellow.po $ mv messages en_US/LC_MESSAGES/guide.mo
Finally, we run the program to test it:
$ gawk -f guide.awk -| Hey man, relax! -| Like, the scoop is 42 -| Pardon me, Zaphod who?
If the three replacement functions for dcgettext()
, dcngettext()
andbindtextdomain()
(see I18N Portability)are in a file namedlibintl.awk,then we can run guide.awk unchanged as follows:
$ gawk --posix -f guide.awk -f libintl.awk -| Don't Panic -| The Answer Is 42 -| Pardon me, Zaphod who?
gawk itself has been internationalizedusing the GNUgettext
package. (GNU gettext
is described incomplete detail inGNU gettext tools.) As of this writing, the latest version of GNUgettext
isversion 0.18.1.
If a translation of gawk's messages exists,thengawk produces usage messages, warnings,and fatal errors in the local language.
Write documentation as if whoever reads it isa violent psychopath who knows where you live.
Steve English, as quoted by Peter Langston
This chapter discusses advanced features in gawk. It's a bit of a “grab bag” of items that are otherwise unrelatedto each other. First, a command-line option allowsgawk to recognizenondecimal numbers in input data, not just inawkprograms. Then, gawk's special features for sorting arrays are presented. Next, two-way I/O, discussed briefly in earlier parts of thisWeb page, is described in full detail, along with the basicsof TCP/IP networking. Finally, gawkcanprofile an awk program, making it possible to tuneit for performance.
Dynamic Extensions,discusses the ability to dynamically add new built-in functions togawk. As this feature is still immature and likely to change,its description is relegated to an appendix.
If you run gawk with the --non-decimal-data option,you can have nondecimal constants in your input data:
$ echo 0123 123 0x123 | > gawk --non-decimal-data '{ printf "%d, %d, %d\n", > $1, $2, $3 }' -| 83, 123, 291
For this feature to work, write your program so thatgawk treats your data as numeric:
$ echo 0123 123 0x123 | gawk '{ print $1, $2, $3 }' -| 0123 123 0x123
The print
statement treats its expressions as strings. Although the fields can act as numbers when necessary,they are still strings, soprint
does not try to treat themnumerically. You may need to add zero to a field to force it tobe treated as a number. For example:
$ echo 0123 123 0x123 | gawk --non-decimal-data ' > { print $1, $2, $3 > print $1 + 0, $2 + 0, $3 + 0 }' -| 0123 123 0x123 -| 83 123 291
Because it is common to have decimal data with leading zeros, and becauseusing this facility could lead to surprising results, the default is to leave itdisabled. If you want it, you must explicitly request it.
CAUTION:
Use of this option is not recommended.It can break old programs very badly. Instead, use the
strtonum()
function to convert your data(see Nondecimal-numbers). This makes your programs easier to write and easier to read, andleads to less surprising results.
gawk lets you control the order in which ‘for (i in array)’ loopswill traverse an array.
In addition, two built-in functions, asort()
and asorti()
,let you sort arrays based on the array values and indices, respectively. These two functions also provide control over the sorting criteria usedto order the elements during sorting.
By default, the order in which a ‘for (i in array)’ loopscans an array is not defined; it is generally based uponthe internal implementation of arrays insideawk.
Often, though, it is desirable to be able to loop over the elementsin a particular order that you, the programmer, choose.gawklets you do this; this subsection describes how.
The value of PROCINFO["sorted_in"]
can be a function name. This lets you traverse an array based on any custom criterion. The array elements are ordered according to the return value of thisfunction. The comparison function should be defined with at leastfour arguments:
function comp_func(i1, v1, i2, v2) { compare elements 1 and 2 in some fashion return < 0; 0; or > 0 }
Here, i1 and i2 are the indices, and v1 and v2are the corresponding values of the two elements being compared. Either v1 or v2, or both, can be arrays if the array beingtraversed contains subarrays as values. The three possible return valuesare interpreted this way:
comp_func(i1, v1, i2, v2) < 0
comp_func(i1, v1, i2, v2) == 0
comp_func(i1, v1, i2, v2) > 0
Our first comparison function can be used to scan an array innumerical order of the indices:
function cmp_num_idx(i1, v1, i2, v2) { # numerical index comparison, ascending order return (i1 - i2) }
Our second function traverses an array based on the string order ofthe element values rather than by indices:
function cmp_str_val(i1, v1, i2, v2) { # string value comparison, ascending order v1 = v1 "" v2 = v2 "" if (v1 < v2) return -1 return (v1 != v2) }
The thirdcomparison function makes all numbers, and numeric strings withoutany leading or trailing spaces, come out first during loop traversal:
function cmp_num_str_val(i1, v1, i2, v2, n1, n2) { # numbers before string value comparison, ascending order n1 = v1 + 0 n2 = v2 + 0 if (n1 == v1) return (n2 == v2) ? (n1 - n2) : -1 else if (n2 == v2) return 1 return (v1 < v2) ? -1 : (v1 != v2) }
Here is a main program to demonstrate how gawkbehaves using each of the previous functions:
BEGIN { data["one"] = 10 data["two"] = 20 data[10] = "one" data[100] = 100 data[20] = "two" f[1] = "cmp_num_idx" f[2] = "cmp_str_val" f[3] = "cmp_num_str_val" for (i = 1; i <= 3; i++) { printf("Sort function: %s\n", f[i]) PROCINFO["sorted_in"] = f[i] for (j in data) printf("\tdata[%s] = %s\n", j, data[j]) print "" } }
Here are the results when the program is run:
$ gawk -f compdemo.awk -| Sort function: cmp_num_idx Sort by numeric index -| data[two] = 20 -| data[one] = 10 Both strings are numerically zero -| data[10] = one -| data[20] = two -| data[100] = 100 -| -| Sort function: cmp_str_val Sort by element values as strings -| data[one] = 10 -| data[100] = 100 String 100 is less than string 20 -| data[two] = 20 -| data[10] = one -| data[20] = two -| -| Sort function: cmp_num_str_val Sort all numbers before all strings -| data[one] = 10 -| data[two] = 20 -| data[100] = 100 -| data[10] = one -| data[20] = two
Consider sorting the entries of a GNU/Linux system password fileaccording to login names. The following program sorts recordsby a specific field position and can be used for this purpose:
# sort.awk --- simple program to sort by field position # field position is specified by the global variable POS function cmp_field(i1, v1, i2, v2) { # comparison by value, as string, and ascending order return v1[POS] < v2[POS] ? -1 : (v1[POS] != v2[POS]) } { for (i = 1; i <= NF; i++) a[NR][i] = $i } END { PROCINFO["sorted_in"] = "cmp_field" if (POS < 1 || POS > NF) POS = 1 for (i in a) { for (j = 1; j <= NF; j++) printf("%s%c", a[i][j], j < NF ? ":" : "") print "" } }
The first field in each entry of the password file is the user's login name,and the fields are seperated by colons. Each record defines a subarray (seeArrays of Arrays),with each field as an element in the subarray. Running the program produces thefollowing output:
$ gawk -vPOS=1 -F: -f sort.awk /etc/passwd -| adm:x:3:4:adm:/var/adm:/sbin/nologin -| apache:x:48:48:Apache:/var/www:/sbin/nologin -| avahi:x:70:70:Avahi daemon:/:/sbin/nologin ...
The comparison should normally always return the same value when given aspecific pair of array elements as its arguments. If inconsistentresults are returned then the order is undefined. This behavior can beexploited to introduce random order into otherwise seeminglyordered data:
function cmp_randomize(i1, v1, i2, v2) { # random order return (2 - 4 * rand()) }
As mentioned above, the order of the indices is arbitrary if twoelements compare equal. This is usually not a problem, but lettingthe tied elements come out in arbitrary order can be an issue, especiallywhen comparing item values. The partial ordering of the equal elementsmay change during the next loop traversal, if other elements are added orremoved from the array. One way to resolve ties when comparing elementswith otherwise equal values is to include the indices in the comparisonrules. Note that doing this may make the loop traversal less efficient,so consider it only if necessary. The following comparison functionsforce a deterministic order, and are based on the fact that theindices of two elements are never equal:
function cmp_numeric(i1, v1, i2, v2) { # numerical value (and index) comparison, descending order return (v1 != v2) ? (v2 - v1) : (i2 - i1) } function cmp_string(i1, v1, i2, v2) { # string value (and index) comparison, descending order v1 = v1 i1 v2 = v2 i2 return (v1 > v2) ? -1 : (v1 != v2) }
A custom comparison function can often simplify ordered looptraversal, and the sky is really the limit when it comes todesigning such a function.
When string comparisons are made during a sort, either for elementvalues where one or both aren't numbers, or for element indiceshandled as strings, the value ofIGNORECASE
(see Built-in Variables) controls whetherthe comparisons treat corresponding uppercase and lowercase letters asequivalent or distinct.
Another point to keep in mind is that in the case of subarraysthe element values can themselves be arrays; a production comparisonfunction should use theisarray()
function(see Type Functions),to check for this, and choose a defined sorting order for subarrays.
All sorting based on PROCINFO["sorted_in"]
is disabled in POSIX mode,since thePROCINFO
array is not special in that case.
As a side note, sorting the array indices before traversingthe array has been reported to add 15% to 20% overhead to theexecution time ofawk programs. For this reason,sorted array traversal is not the default.
As described inControlling Scanning With A Function,you can provide the name of a function as the value ofPROCINFO["sorted_in"]
to specify custom sorting criteria.
Often, though, you may wish to do something simple, such as“sort based on comparing the indices in ascending order,”or “sort based on comparing the values in descending order.”Having to write a simple comparison function for this purposefor use in all of your programs becomes tedious. For the common simple cases, gawk providesthe option of supplying special names that do the requestedsorting for you. You can think of them as “predefined” sorting functions,if you like, although the names purposely include charactersthat are not valid in real awk function names.
The following special values are available:
"@ind_str_asc"
"10"
rather than numeric 10.)
"@ind_num_asc"
"@val_type_asc"
"@val_str_asc"
"@val_num_asc"
qsort()
function.
59
"@ind_str_desc"
"@ind_num_desc"
"@val_type_desc"
"@val_str_desc"
"@val_num_desc"
"@unsorted"
"sorted_in"
element from the
PROCINFO
array,if it previously had a value assigned to it.
The array traversal order is determined before the for
loopstarts to run. ChangingPROCINFO["sorted_in"]
in the loop bodywill not affect the loop.
For example:
$ gawk 'BEGIN { > a[4] = 4 > a[3] = 3 > for (i in a) > print i, a[i] > }' -| 4 4 -| 3 3 $ gawk 'BEGIN { > PROCINFO["sorted_in"] = "@str_ind_asc" > a[4] = 4 > a[3] = 3 > for (i in a) > print i, a[i] > }' -| 3 3 -| 4 4
When sorting an array by element values, if a value happens to bea subarray then it is considered to be greater than any string ornumeric value, regardless of what the subarray itself contains,and all subarrays are treated as being equal to each other. Theirorder relative to each other is determined by their index strings.
In most awk implementations, sorting an array requireswriting asort
function. While this can be educational for exploring different sorting algorithms,usually that's not the point of the program.gawk provides the built-in asort()
andasorti()
functions(see String Functions)for sorting arrays. For example:
populate the array data n = asort(data) for (i = 1; i <= n; i++) do something with data[i]
After the call to asort()
, the array data
is indexed from 1to some numbern, the total number of elements in data
. (This count is asort()
's return value.) data[1]
<= data[2]
<= data[3]
, and so on. The array elements are compared as strings.
An important side effect of callingasort()
is thatthe array's original indices are irrevocably lost. As this isn't always desirable,asort()
accepts asecond argument:
populate the array source n = asort(source, dest) for (i = 1; i <= n; i++) do something with dest[i]
In this case, gawk copies the source
array into thedest
array and then sorts dest
, destroying its indices. However, thesource
array is not affected.
asort()
accepts a third string argumentto control comparison of array elements. As withPROCINFO["sorted_in"]
, this argument may be thename of a user-defined function, or one of the predefined namesthatgawk provides(see Controlling Scanning With A Function).
NOTE: In all cases, the sorted element values consist of the originalarray's element values. The ability to control comparison merelyaffects the way in which they are sorted.
Often, what's needed is to sort on the values of the indicesinstead of the values of the elements. To do that, use theasorti()
function. The interface is identical to that ofasort()
, except that the index values are used for sorting, andbecome the values of the result array:
{ source[$0] = some_func($0) } END { n = asorti(source, dest) for (i = 1; i <= n; i++) { Work with sorted indices directly: do something with dest[i] ... Access original array via sorted indices: do something with source[dest[i]] } }
Similar to asort()
,in all cases, the sorted element values consist of the originalarray's indices. The ability to control comparison merelyaffects the way in which they are sorted.
Sorting the array by replacing the indices provides maximal flexibility. To traverse the elements in decreasing order, use a loop that goes fromn down to 1, either over the elements or over the indices.60
Copying array indices and elements isn't expensive in terms of memory. Internally,gawk maintains reference counts to data. For example, whenasort()
copies the first array to the second one,there is only one copy of the original array elements' data, even thoughboth arrays use the values.
BecauseIGNORECASE
affects string comparisons, the valueof IGNORECASE
also affects sorting for bothasort()
and asorti()
. Note also that the locale's sorting order doesnotcome into play; comparisons are based on character values only.61Caveat Emptor.
From: [email protected] (Mike Brennan) Newsgroups: comp.lang.awk Subject: Re: Learn the SECRET to Attract Women Easily Date: 4 Aug 1997 17:34:46 GMT Message-ID: <[email protected]> <!-- References: <5s20dn 2e1="" chronicle="" concentric="" net=""> --> On 3 Aug 1997 13:17:43 GMT, Want More Dates??? <[email protected]> wrote: >Learn the SECRET to Attract Women Easily > >The SCENT(tm) Pheromone Sex Attractant For Men to Attract Women The scent of awk programmers is a lot more attractive to women than the scent of perl programmers. -- Mike Brennan
It is often useful to be able tosend data to a separate program forprocessing and then read the result. This can always bedone with temporary files:
# Write the data for processing tempfile = ("mydata." PROCINFO["pid"]) while (not done with data) print data | ("subprogram > " tempfile) close("subprogram > " tempfile) # Read the results, remove tempfile when done while ((getline newdata < tempfile) > 0) process newdata appropriately close(tempfile) system("rm " tempfile)
This works, but not elegantly. Among other things, it requires thatthe program be run in a directory that cannot be shared among users;for example,/tmp will not do, as another user might happento be using a temporary file with the same name.
However, with gawk, it is possible toopen a two-way pipe to another process. The second process istermed a coprocess, since it runs in parallel withgawk. The two-way connection is created using the ‘|&’ operator(borrowed from the Korn shell,ksh):62
do { print data |& "subprogram" "subprogram" |& getline results } while (data left to process) close("subprogram")
The first time an I/O operation is executed using the ‘|&’operator,gawk creates a two-way pipeline to a child processthat runs the other program. Output created withprint
or printf
is written to the program's standard input, andoutput from the program's standard output can be read by thegawkprogram using getline
. As is the case with processes started by ‘|’, the subprogramcan be any program, or pipeline of programs, that can be started bythe shell.
There are some cautionary items to be aware of:
getline
in order to readthe coprocess's results. This could lead to a situationknown asdeadlock, where each process is waiting for theother one to do something. It is possible to close just one end of the two-way pipe toa coprocess, by supplying a second argument to theclose()
function of either "to"
or "from"
(seeClose Files And Pipes). These strings tell gawk to close the end of the pipethat sends data to the coprocess or the end that reads from it,respectively.
This is particularly necessary in order to usethe systemsort utility as part of a coprocess;sort must readall of its inputdata before it can produce any output. The sort program does not receive an end-of-file indicationuntilgawk closes the write end of the pipe.
When you have finished writing data to the sortutility, you can close the"to"
end of the pipe, andthen start reading sorted data via getline
. For example:
BEGIN { command = "LC_ALL=C sort" n = split("abcdefghijklmnopqrstuvwxyz", a, "") for (i = n; i > 0; i--) print a[i] |& command close(command, "to") while ((command |& getline line) > 0) print "got", line close(command) }
This program writes the letters of the alphabet in reverse order, oneper line, down the two-way pipe tosort. It then closes thewrite end of the pipe, so thatsort receives an end-of-fileindication. This causessort to sort the data and write thesorted data back to thegawk program. Once all of the datahas been read,gawk terminates the coprocess and exits.
As a side note, the assignment ‘LC_ALL=C’ in thesortcommand ensures traditional Unix (ASCII) sorting fromsort.
You may also use pseudo-ttys (ptys) fortwo-way communication instead of pipes, if your system supports them. This is done on a per-command basis, by setting a special elementin the PROCINFO
array(see Auto-set),like so:
command = "sort -nr" # command, save in convenience variable PROCINFO[command, "pty"] = 1 # update PROCINFO print ... |& command # start two-way pipe ...
Using ptys avoids the buffer deadlock issues described earlier, at someloss in performance. If your system does not have ptys, or if all thesystem's ptys are in use,gawk automatically falls back tousing regular pipes.
EMISTERED
:
A host is a host from coast to coast,
and no-one can talk to host that's close,
unless the host that isn't close
is busy hung or dead.
In addition to being able to open a two-way pipeline to a coprocesson the same system(seeTwo-way I/O),it is possible to make a two-way connection toanother process on another system across an IP network connection.
You can think of this as just a very long two-way pipeline toa coprocess. The waygawk decides that you want to use TCP/IP networking isby recognizing special file names that begin with one of ‘/inet/’,‘/inet4/’ or ‘/inet6’.
The full syntax of the special file name is/net-type/protocol/local-port/remote-host/remote-port. The components are:
getaddrinfo()
function.
NOTE: Failure in opening a two-way socket will result in a non-fatal errorbeing returned to the calling code. The value of
ERRNO
indicatesthe error (see Auto-set).
Consider the following very simple example:
BEGIN { Service = "/inet/tcp/0/localhost/daytime" Service |& getline print $0 close(Service) }
This program reads the current date and time from the local system'sTCP ‘daytime’ server. It then prints the results and closes the connection.
Because this topic is extensive, the use of gawk forTCP/IP programming is documented separately. SeeTCP/IP Internetworking with gawk,which comes as part of thegawk distribution,for a much more complete introduction and discussion, as well asextensive examples.
You may produce executiontraces of your awk programs. This is done with a specially compiled version ofgawk,called pgawk (“profilinggawk”).
pgawk is identical in every way to gawk, except that whenit has finished running, it creates a profile of your program in a filenamedawkprof.out. Because it is profiling, it also executes up to 45% slower thangawk normally does.
As shown in the following example,the--profile option can be used to change the name of the filewherepgawk will write the profile:
pgawk --profile=myprog.prof -f myprog.awk data1 data2
In the above example, pgawk places the profile inmyprog.prof instead of inawkprof.out.
Here is a samplesession showing a simple awk program, its input data, and theresults from runningpgawk. First, the awk program:
BEGIN { print "First BEGIN rule" } END { print "First END rule" } /foo/ { print "matched /foo/, gosh" for (i = 1; i <= 3; i++) sing() } { if (/foo/) print "if is true" else print "else is true" } BEGIN { print "Second BEGIN rule" } END { print "Second END rule" } function sing( dummy) { print "I gotta be me!" }
Following is the input data:
foo bar baz foo junk
Here is the awkprof.out that results from runningpgawkon this program and data (this example also illustrates thatawkprogrammers sometimes have to work late):
# gawk profile, created Sun Aug 13 00:00:15 2000 # BEGIN block(s) BEGIN { 1 print "First BEGIN rule" 1 print "Second BEGIN rule" } # Rule(s) 5 /foo/ { # 2 2 print "matched /foo/, gosh" 6 for (i = 1; i <= 3; i++) { 6 sing() } } 5 { 5 if (/foo/) { # 2 2 print "if is true" 3 } else { 3 print "else is true" } } # END block(s) END { 1 print "First END rule" 1 print "Second END rule" } # Functions, listed alphabetically 6 function sing(dummy) { 6 print "I gotta be me!" }
This example illustrates many of the basic features of profiling output. They are as follows:
BEGIN
rule,BEGINFILE
rule,pattern/action rules,ENDFILE
rule,END
rule and functions, listedalphabetically. Multiple BEGIN
andEND
rules are merged together,as are multiple BEGINFILE
andENDFILE
rules. if
-else
statement shows how many timesthe condition was tested. To the right of the opening left brace for theif
's bodyis a count showing how many times the condition was true. The count for theelse
indicates how many times the test failed. for
or while
) shows how many times the loop test was executed. (Because of this, you can't just look at the count on the firststatement in a rule to determine how many times the rule was executed. If the first statement is a loop, the count is misleading.) function
keyword indicates how many times the function was called. The counts next to the statements in the body show how many timesthose statements were executed. if
, else
, or loop is only a single statement. print
and printf
only whentheprint
or printf
statement is followed by a redirection. Similarly, ifthe target of a redirection isn't a scalar, it gets parenthesized.BEGIN
and END
rules,the pattern/action rules, and the functions. The profiled version of your program may not look exactly like what youtyped when you wrote it. This is becausepgawk creates theprofiled version by “pretty printing” its internal representation ofthe program. The advantage to this is thatpgawk can producea standard representation. The disadvantage is that all source-codecomments are lost, as are the distinctions among multipleBEGIN
,END
, BEGINFILE
, and ENDFILE
rules. Also, things such as:
/foo/
come out as:
/foo/ { print $0 }
which is correct, but possibly surprising.
Besides creating profiles when a program has completed,pgawk can produce a profile while it is running. This is useful if your