Studying note of GCC-3.4.6 source (74)

5.3.1.2.    Macro invocation

For our case, this next token is “FUN”. For this node, its type of course is NT_MACRO. As long as its not disable macro and not in the environment that prevent macro expansion (e.g., in handling assertion, pragma, #if series directives; parsing marco argument, function-like marco invocation), enter_macro_context will be invoked.

 

709  static int

710  enter_macro_context (cpp_reader *pfile, cpp_hashnode *node)                      in cppmacro.c

711   {

712    /* The presence of a macro invalidates a file's controlling macro.  */

713    pfile->mi_valid = false;

714 

715    pfile->state.angled_headers = false;

716 

717   /* Handle standard macros.  */

718    if (! (node->flags & NODE_BUILTIN))

719    {

720      cpp_macro *macro = node->value.macro;

721 

722      if (macro->fun_like)

723      {

724        _cpp_buff *buff;

725 

726        pfile->state.prevent_expansion++;

727        pfile->keep_tokens++;

728        pfile->state.parsing_args = 1;

729        buff = funlike_invocation_p (pfile, node);

730        pfile->state.parsing_args = 0;

731        pfile->keep_tokens--;

732        pfile->state.prevent_expansion--;

733 

734        if (buff == NULL)

735        {

736          if (CPP_WTRADITIONAL (pfile) && ! node->value.macro->syshdr)

737            cpp_error (pfile, CPP_DL_WARNING,

738               "function-like macro /"%s/" must be used with arguments in traditional C",

739                   NODE_NAME (node));

740 

741          return 0;

742        }

743 

744        if (macro->paramc > 0)

745          replace_args (pfile, node, macro, (macro_arg *) buff->base);

746        _cpp_release_buff (pfile, buff);

747      }

748 

749      /* Disable the macro within its expansion.  */

750      node->flags |= NODE_DISABLED;

751 

752      macro->used = 1;

753 

754      if (macro->paramc == 0)

755        push_token_context (pfile, node, macro->exp.tokens, macro->count);

756 

757      return 1;

758    }

759 

760   /* Handle built-in macros and the _Pragma operator.  */

761    return builtin_macro (pfile, node);

762  }

 

Pay attention to the setting of pfile before and after the call of funlike_invocation_p at line 729.

5.3.1.2.1.            Funciton-like macro expansion – arguments collection

For non-builtin macro, it needs setup context to take place the macro expansion. At line 720, remember that the whole definition is saved into macro field in the identifier node of the macro (see create_iso_definition). For non-function-like macro, put these tokens into a new context is OK, as line 755 above does. But for function-like macro, it needs macro argument replacement, in which  the new context is created. Note that at line 728, slot parsing_args is set as 1 before invoking funlik_invocation_p, and reset to 0 after it.

 

669  static _cpp_buff *

670  funlike_invocation_p (cpp_reader *pfile, cpp_hashnode *node)              in cppmacro.c

671  {

672    const cpp_token *token, *padding = NULL;

673 

674    for (;;)

675    {

676      token = cpp_get_token (pfile);

677      if (token->type != CPP_PADDING)

678        break;

679      if (padding == NULL

680          || (!(padding->flags & PREV_WHITE) && token->val.source == NULL))

681        padding = token;

682    }

683 

684    if (token->type == CPP_OPEN_PAREN)

685    {

686      pfile->state.parsing_args = 2;

687      return collect_args (pfile, node);

688    }

689 

690    /* CPP_EOF can be the end of macro arguments, or the end of the

691      file. We mustn't back up over the latter. Ugh.  */

692    if (token->type != CPP_EOF || token == &pfile->eof)

693    {

694      /* Back up. We may have skipped padding, in which case backing

695        up more than one token when expanding macros is in general

696        too difficult. We re-insert it in its own context.  */

697      _cpp_backup_tokens (pfile, 1);

698      if (padding)

699        push_token_context (pfile, NULL, padding, 1);

700    }

701 

702    return NULL;

703  }

 

Routine collect_args collects the real arguments for the macro invocation. These arguments are saved for later replacement. Note that before entering the function, field parsing_args is set as 2 – open parentheses is seen.

 

535  static _cpp_buff *

536  collect_args (cpp_reader *pfile, const cpp_hashnode *node)                   in cppmacro.c

537  {

538    _cpp_buff *buff, *base_buff;

539    cpp_macro *macro;

540    macro_arg *args, *arg;

541    const cpp_token *token;

542    unsigned int argc;

543 

544    macro = node->value.macro;

545    if (macro->paramc)

546      argc = macro->paramc;

547    else

548      argc = 1;

549    buff = _cpp_get_buff (pfile, argc * (50 * sizeof (cpp_token *)

550                              + sizeof (macro_arg)));

551    base_buff = buff;

552    args = (macro_arg *) buff->base;

553    memset (args, 0, argc * sizeof (macro_arg));

554    buff->cur = (unsigned char *) &args[argc];

555    arg = args, argc = 0;

 

Slot paramc in cpp_macro records the number of arguments of the macro. Type macro_arg is designed to hold necessary information for the argument replacement.

 

32    struct macro_arg                                                                                 in cppmacro.c

33    {

34      const cpp_token **first;    /* First token in unexpanded argument.  */

35      const cpp_token **expanded;    /* Macro-expanded argument.  */

36      const cpp_token *stringified;    /* Stringified argument.  */

37      unsigned int count;           /* # of tokens in argument.  */

38      unsigned int expanded_count;   /* # of tokens in expanded argument.  */

39    };

 

Below, see cpp_get_token is invoked at line 578 to fetch the tokens of argument. For our case, directive of #ifdef will be read in as the third token, and according to [12] the behavor of the directive is undefined by standard, but GCC decides to let those directives behave as normal – that is invoking _cpp_handle_directive. Note, during parsing arguments, argument being macro itself is prohibited from expansion as defined by standards (but it is expanded in argument replacement).

 

collect_args (continue)

 

557    /* Collect the tokens making up each argument. We don't yet know

558      how many arguments have been supplied, whether too many or too

559      few. Hence the slightly bizarre usage of "argc" and "arg".  */

560    do

561    {

562      unsigned int paren_depth = 0;

563      unsigned int ntokens = 0;

564 

565      argc++;

566      arg->first = (const cpp_token **) buff->cur;

567 

568      for (;;)

569      {

570        /* Require space for 2 new tokens (including a CPP_EOF).  */

571        if ((unsigned char *) &arg->first[ntokens + 2] > buff->limit)

572        {

573          buff = _cpp_append_extend_buff (pfile, buff,

574                                      1000 * sizeof (cpp_token *));

575          arg->first = (const cpp_token **) buff->cur;

576        }

577 

578        token = cpp_get_token (pfile);

579 

580        if (token->type == CPP_PADDING)

581        {

582          /* Drop leading padding.  */

583          if (ntokens == 0)

584            continue;

585        }

586        else if (token->type == CPP_OPEN_PAREN)

587          paren_depth++;

588        else if (token->type == CPP_CLOSE_PAREN)

589        {

590          if (paren_depth-- == 0)

591            break;

592        }

593        else if (token->type == CPP_COMMA)

594        {

595          /* A comma does not terminate an argument within

596            parentheses or as part of a variable argument.  */

597          if (paren_depth == 0

598             && ! (macro->variadic && argc == macro->paramc))

599            break;

600        }

601        else if (token->type == CPP_EOF

602             || (token->type == CPP_HASH && token->flags & BOL))

603          break;

604 

605        arg->first[ntokens++] = token;

606      }

607 

608      /* Drop trailing padding.  */

609      while (ntokens > 0 && arg->first[ntokens - 1]->type == CPP_PADDING)

610        ntokens--;

611  

612      arg->count = ntokens;

613      arg->first[ntokens] = &pfile->eof;

614 

615      /* Terminate the argument. Excess arguments loop back and

616        overwrite the final legitimate argument, before failing.  */

617      if (argc <= macro->paramc)

618      {

619        buff->cur = (unsigned char *) &arg->first[ntokens + 1];

620        if (argc != macro->paramc)

621          arg++;

622      }

623    }

624    while (token->type != CPP_CLOSE_PAREN && token->type != CPP_EOF);

 

Notice that even arguments are separated by comma, but ISO standard allows arbitrary number of whitespaces inserted around the comma, so it uses nested loops to handle the case. The inner FOR loop retrieves the may having tailing paddings argument (see that the paddings are also tokens), then while loop at line 609 strips these padding tokens, leaves the argument.

 

collect_args (continue)

 

626    if (token->type == CPP_EOF)

627    {

628      /* We still need the CPP_EOF to end directives, and to end

629        pre-expansion of a macro argument. Step back is not

630        unconditional, since we don't want to return a CPP_EOF to our

631        callers at the end of an -include-d file.  */

632      if (pfile->context->prev || pfile->state.in_directive)

633        _cpp_backup_tokens (pfile, 1);

634      cpp_error (pfile, CPP_DL_ERROR,

635               "unterminated argument list invoking macro /"%s/"",

636               NODE_NAME (node));

637    }

638    else

639    {

640      /* A single empty argument is counted as no argument.  */

641      if (argc == 1 && macro->paramc == 0 && args[0].count == 0)

642        argc = 0;

643      if (_cpp_arguments_ok (pfile, macro, node, argc))

644      {

645        /* GCC has special semantics for , ## b where b is a varargs

646          parameter: we remove the comma if b was omitted entirely.

647          If b was merely an empty argument, the comma is retained.

648          If the macro takes just one (varargs) parameter, then we

649          retain the comma only if we are standards conforming.

650 

651          If FIRST is NULL replace_args () swallows the comma.  */

652        if (macro->variadic && (argc < macro->paramc

653                            || (argc == 1 && args[0].count == 0

654                              && !CPP_OPTION (pfile, std))))

655          args[macro->paramc - 1].first = NULL;

656        return base_buff;

657      }

658    }

659 

660   /* An error occurred.  */

661    _cpp_release_buff (pfile, base_buff);

662    return NULL;

663  }

 

As abnormal case, end of file is encountered before closing parenthesis is found. Otherwise, after closing parenthesis finishes the retrieving, the arguments themselves need be validated.

 

494  bool

495  _cpp_arguments_ok(cpp_reader *pfile, cpp_macro *macro, const cpp_hashnode *node, unsigned int argc)

496  {

497    if (argc == macro->paramc)

498      return true;

499 

500    if (argc < macro->paramc)

501    {

502      /* As an extension, a rest argument is allowed to not appear in

503        the invocation at all.

504        e.g. #define debug(format, args...) something

505          debug("string");

506 

507        This is exactly the same as if there had been an empty rest

508        argument - debug("string", ).  */

509 

510      if (argc + 1 == macro->paramc && macro->variadic)

511       {

512        if (CPP_PEDANTIC (pfile) && ! macro->syshdr)

513          cpp_error (pfile, CPP_DL_PEDWARN,

514                   "ISO C99 requires rest arguments to be used");

515        return true;

516      }

517 

518      cpp_error (pfile, CPP_DL_ERROR,

519               "macro /"%s/" requires %u arguments, but only %u given",

520                NODE_NAME (node), macro->paramc, argc);

521    }

522    else

523      cpp_error (pfile, CPP_DL_ERROR,

524               "macro /"%s/" passed %u arguments, but takes just %u",

525               NODE_NAME (node), argc, macro->paramc);

526 

527    return false;

528  }

 

The validation only concerns the matching of number of arguments.

5.3.1.2.2.            Funciton-like macro expansion – arguments replacement

After collecting the parameters during the invocation, at line 745 in enter_macro_context, these parameters will replace the declared arguments in definition body by replace_args.

 

768  static void

769  replace_args(cpp_reader *pfile, cpp_hashnode *node, cpp_macro *macro, macro_arg *args)

770  {

771    unsigned int i, total;

772    const cpp_token *src, *limit;

773    const cpp_token **dest, **first;

774    macro_arg *arg;

775    _cpp_buff *buff;

776 

777    /* First, fully macro-expand arguments, calculating the number of

778      tokens in the final expansion as we go. The ordering of the if

779      statements below is subtle; we must handle stringification before

780      pasting.  */

781    total = macro->count;

782    limit = macro->exp.tokens + macro->count;

783 

784    for (src = macro->exp.tokens; src < limit; src++)

785      if (src->type == CPP_MACRO_ARG)

786      {

787        /* Leading and trailing padding tokens.  */

788        total += 2;

789 

790        /* We have an argument. If it is not being stringified or

791          pasted it is macro-replaced before insertion.  */

792        arg = &args[src->val.arg_no - 1];

793 

794        if (src->flags & STRINGIFY_ARG)

795        {

796          if (!arg->stringified)

797           arg->stringified = stringify_arg (pfile, arg);

798        }

799        else if ((src->flags & PASTE_LEFT)

800               || (src > macro->exp.tokens && (src[-1].flags & PASTE_LEFT)))

801         total += arg->count - 1;

802        else

803        {

804          if (!arg->expanded)

805            expand_arg (pfile, arg);

806          total += arg->expanded_count - 1;

807        }

808      }

 

At line 781 slot count records the number of tokens in the definition body, but with replacement, the number may not correct. FOR loop beginning at line 784 updates this number. For those tokens of argument in definition body, they are of type CPP_MACRO_ARG.

According to ISO C/C++, token following ‘#’ (the stringify operator) and tokens around ‘##’ (the paste operator) will not be expanded even they are macro themselves. For stringify case, one token is allowed after ‘#’, so no token number change. For paste case, in fact, we can pass an expression for the argument, for example:

 

#define fA(A, B)  c = A##B;

fA (a+b, B)     // output a+bB;

 

So the number of tokens added is the argument’s token number minus 1. While for the left case, argument being macro itself should be expanded normally.

 

979  static void

980  expand_arg (cpp_reader *pfile, macro_arg *arg)                                   in cppmacro.c

981  {

982    unsigned int capacity;

983    bool saved_warn_trad;

984 

985    if (arg->count == 0)

986      return;

987 

988    /* Don't warn about funlike macros when pre-expanding.  */

989    saved_warn_trad = CPP_WTRADITIONAL (pfile);

990    CPP_WTRADITIONAL (pfile) = 0;

991 

992    /* Loop, reading in the arguments.  */

993    capacity = 256;

994    arg->expanded = xmalloc (capacity * sizeof (cpp_token *));

995 

996    push_ptoken_context (pfile, NULL, NULL, arg->first, arg->count + 1);

997    for (;;)

998    {

999      const cpp_token *token;

1000

1001     if (arg->expanded_count + 1 >= capacity)

1002     {

1003       capacity *= 2;

1004       arg->expanded = xrealloc (arg->expanded,

1005                             capacity * sizeof (cpp_token *));

1006     }

1007

1008     token = cpp_get_token (pfile);

1009

1010     if (token->type == CPP_EOF)

1011      break;

1012

1013     arg->expanded[arg->expanded_count++] = token;

1014   }

1015

1016   _cpp_pop_context (pfile);

1017

1018   CPP_WTRADITIONAL (pfile) = saved_warn_trad;

1019 }

 

To parsing arbitrary nested macro correctly, it is necessary to setup parsing context as push_ptoken_context does.

 

931  static void

932  push_ptoken_context (cpp_reader *pfile, cpp_hashnode *macro,             in cppmacro.c

933              _cpp_buff *buff, const cpp_token **first, unsigned int count)

934  {

935    cpp_context *context = next_context (pfile);

936 

937    context->direct_p = false;

938    context->macro = macro;

939    context->buff = buff;

940    FIRST (context).ptoken = first;

941    LAST (context).ptoken = first + count;

942  }

 

After setting up the context, in expand_arg at line 1008, cpp_get_token next fetches token from the context. If the argument is macro too, enter_macro_context at line 1099 will be invoked again. Telling from the invocation sequence, the handling of nested macros are depth first, so when exit FOR loop at line 1011, all macros nested within must be handled. Even note that for this expanded argument, the tokens generated are saved within expanded field of corresponding object of macro_arg.

 

replace_args (continue)

 

810    /* Now allocate space for the expansion, copy the tokens and replace

811       the arguments.  */

812    buff = _cpp_get_buff (pfile, total * sizeof (cpp_token *));

813    first = (const cpp_token **) buff->base;

814    dest = first;

815 

816    for (src = macro->exp.tokens; src < limit; src++)

817    {

818      unsigned int count;

819      const cpp_token **from, **paste_flag;

820 

821      if (src->type != CPP_MACRO_ARG)

822      {

823        *dest++ = src;

824        continue;

825      }

826 

827      paste_flag = 0;

828      arg = &args[src->val.arg_no - 1];

829      if (src->flags & STRINGIFY_ARG)

830        count = 1, from = &arg->stringified;

831      else if (src->flags & PASTE_LEFT)

832        count = arg->count, from = arg->first;

833      else if (src != macro->exp.tokens && (src[-1].flags & PASTE_LEFT))

834      {

835        count = arg->count, from = arg->first;

836        if (dest != first)

837        {

838          if (dest[-1]->type == CPP_COMMA

839             && macro->variadic

840             && src->val.arg_no == macro->paramc)

841          {

842            /* Swallow a pasted comma if from == NULL, otherwise

843              drop the paste flag.  */

844            if (from == NULL)

845              dest--;

846            else

847              paste_flag = dest - 1;

848          }

849          /* Remove the paste flag if the RHS is a placemarker.  */

850          else if (count == 0)

851            paste_flag = dest - 1;

852        }

853      }

854      else

855        count = arg->expanded_count, from = arg->expanded;

856 

857      /* Padding on the left of an argument (unless RHS of ##).  */

858      if ((!pfile->state.in_directive || pfile->state.directive_wants_padding)

859          && src != macro->exp.tokens && !(src[-1].flags & PASTE_LEFT))

860        *dest++ = padding_token (pfile, src);

861 

862      if (count)

863      {

864        memcpy (dest, from, count * sizeof (cpp_token *));

865        dest += count;

866 

867        /* With a non-empty argument on the LHS of ##, the last

868          token should be flagged PASTE_LEFT.  */

869        if (src->flags & PASTE_LEFT)

870          paste_flag = dest - 1;

871      }

872 

873      /* Avoid paste on RHS (even case count == 0).  */

874      if (!pfile->state.in_directive && !(src->flags & PASTE_LEFT))

875        *dest++ = &pfile->avoid_paste;

876 

877      /* Add a new paste flag, or remove an unwanted one.  */

878      if (paste_flag)

879      {

880        cpp_token *token = _cpp_temp_token (pfile);

881        token->type = (*paste_flag)->type;

882        token->val.str = (*paste_flag)->val.str;

883        if (src->flags & PASTE_LEFT)

884          token->flags = (*paste_flag)->flags | PASTE_LEFT;

885        else

886          token->flags = (*paste_flag)->flags & ~PASTE_LEFT;

887        *paste_flag = token;

888      }

889    }

890 

891    /* Free the expanded arguments.  */

892    for (i = 0; i < macro->paramc; i++)

893      if (args[i].expanded)

894        free (args[i].expanded);

895 

896    push_ptoken_context (pfile, node, buff, first, dest - first);

897  }

 

As the space needed for expanding the macro has been calculated, a buffer of this size is created and holds the tokens of the expanded macro. And as it is possible to pass a token sequence for an argument, for token paste, it needs mark the first (on right hand side of ##) or the last (on the left hand side of ##) token in the sequence as PASTE_LEFT. A new parsing context is created again at line 896, solely to make them the next fetched tokens.

For paste case, in cpp_get_token, paste_all_tokens will be invoked at line 1071.

 

448  static void

449  paste_all_tokens (cpp_reader *pfile, const cpp_token *lhs)                     in cppmacro.c

450  {

451    const cpp_token *rhs;

452    cpp_context *context = pfile->context;

453 

454    do

455    {

456      /* Take the token directly from the current context. We can do

457        this, because we are in the replacement list of either an

458        object-like macro, or a function-like macro with arguments

459        inserted. In either case, the constraints to #define

460        guarantee we have at least one more token.  */

461      if (context->direct_p)

462        rhs = FIRST (context).token++;

463      else

464        rhs = *FIRST (context).ptoken++;

465 

466      if (rhs->type == CPP_PADDING)

467        abort ();

468 

469      if (!paste_tokens (pfile, &lhs, rhs))

470      {

471        _cpp_backup_tokens (pfile, 1);

472 

473        /* Mandatory error for all apart from assembler.  */

474        if (CPP_OPTION (pfile, lang) != CLK_ASM)

475          cpp_error (pfile, CPP_DL_ERROR,

476                   "pasting /"%s/" and /"%s/" does not give a valid preprocessing token",

477                   cpp_token_as_text (pfile, lhs),

478                   cpp_token_as_text (pfile, rhs));

479        break;

480      }

481    }

482    while (rhs->flags & PASTE_LEFT);

483 

484    /* Put the resulting token in its own context.  */

485    push_token_context (pfile, NULL, lhs, 1);

486  }

 

Note that at line 424, only ‘/’ and ‘=’ can be pasted to generate ‘/=’ operator, other ‘/’ and not ‘=’ pair will have white space inserted between. Then at line 433 below, _cpp_temp_token creates a new token, which has its line NO., column NO., copied fromcur_token of file (e.g., the pasted token). It can ensure that the diagnositc system can give out correct information.

 

407  static bool

408  paste_tokens (cpp_reader *pfile, const cpp_token **plhs, const cpp_token *rhs)

409  {

410    unsigned char *buf, *end;

411     const cpp_token *lhs;

412    unsigned int len;

413    bool valid;

414 

415    lhs = *plhs;

416    len = cpp_token_len (lhs) + cpp_token_len (rhs) + 1;

417    buf = alloca (len);

418    end = cpp_spell_token (pfile, lhs, buf);

419 

420    /* Avoid comment headers, since they are still processed in stage 3.

421      It is simpler to insert a space here, rather than modifying the

422      lexer to ignore comments in some circumstances. Simply returning

423      false doesn't work, since we want to clear the PASTE_LEFT flag.  */

424    if (lhs->type == CPP_DIV && rhs->type != CPP_EQ)

425      *end++ = ' ';

426    end = cpp_spell_token (pfile, rhs, end);

427    *end = '/n';

428 

429    cpp_push_buffer (pfile, buf, end - buf, /* from_stage3 */ true);

430    _cpp_clean_line (pfile);

431 

432    /* Set pfile->cur_token as required by _cpp_lex_direct.  */

433    pfile->cur_token = _cpp_temp_token (pfile);

434    *plhs = _cpp_lex_direct (pfile);

435    valid = pfile->buffer->cur == pfile->buffer->rlimit;

436    _cpp_pop_buffer (pfile);

437 

438    return valid;

439  }

 

Above at line 429, the concatenated literal name is pushed into the buffer slot of file, then following _cpp_lex_direct will fetch this concatenated name as token and assign it to plhs. Also see that when return back to paste_all_tokens, at line 485, this token is placed into a new context too, which will be fetched when next invoke cpp_get_tokenl.

Before leaving this section, we have a look at where token in lookahead slot comes from in _cpp_lex_token. It dues to we step back one or more tokens in base context.

 

1153 void

1154 _cpp_backup_tokens (cpp_reader *pfile, unsigned int count)                  in cppmacro.c

1155 {

1156   if (pfile->context->prev == NULL)

1157   {

1158     pfile->lookaheads += count;

1159     while (count--)

1160     {

1161       pfile->cur_token--;

1162       if (pfile->cur_token == pfile->cur_run->base

1163          /* Possible with -fpreprocessed and no leading #line.  */

1164          && pfile->cur_run->prev != NULL)

1165       {

1166         pfile->cur_run = pfile->cur_run->prev;

1167         pfile->cur_token = pfile->cur_run->limit;

1168       }

1169     }

1170   }

1171   else

1172   {

1173     if (count != 1)

1174       abort ();

1175     if (pfile->context->direct_p)

1176       FIRST (pfile->context).token--;

1177     else

1178       FIRST (pfile->context).ptoken--;

1179   }

1180 }

 

At the end of enter_macro_context, notice that at line 750, now the identifier node of the macro is flagged as expansion disable. That means if the macro is found in the expanded body in later read-in, it will be treated as normal identifier. It prevents macro to invoke itself then cause indefinitely recursion and breakdown the compiler.

And when the context of the macro expansion is handled and popped out, the macro is enable again.

 

1024 void

1025 _cpp_pop_context (cpp_reader *pfile)                                                         in cppmacro.c

1026 {

1027   cpp_context *context = pfile->context;

1028

1029   if (context->macro)

1030     context->macro->flags &= ~NODE_DISABLED;

1031

1032   if (context->buff)

1033     _cpp_release_buff (pfile, context->buff);

1034

1035   pfile->context = context->prev;

1036 }