Namespaces
Variants
Views
Actions

Difference between revisions of "cpp/locale/codecvt"

From cppreference.com
< cpp‎ | locale
(in fact, having looked at the impls, let's be stronger and say single-byte for the locale-independent wchar_t codecvt to avoid further confusion -- this is all re http://stackoverflow.com/questions/42477036)
(Added LWG issue #3767 DR.)
 
(20 intermediate revisions by 10 users not shown)
Line 1: Line 1:
−
{{cpp/title | codecvt}}
+
{{cpp/title|codecvt}}
 
{{cpp/locale/codecvt/navbar}}
 
{{cpp/locale/codecvt/navbar}}
−
{{dcl begin}}
+
{{ddcl|header=locale|
−
{{dcl header | locale}}
+
−
{{dcl | 1=
+
 
template<  
 
template<  
 
     class InternT,  
 
     class InternT,  
 
     class ExternT,  
 
     class ExternT,  
−
     class State
+
     class StateT
 
> class codecvt;
 
> class codecvt;
 
}}
 
}}
−
{{dcl end}}
 
  
−
Class {{lc|std::codecvt}} encapsulates conversion of character strings, including wide and multibyte, from one encoding to another. All file I/O operations performed through {{c|std::basic_fstream<CharT>}} use the {{c|std::codecvt<CharT, char, std::mbstate_t>}} facet of the locale imbued in the stream.
+
Class template {{tt|std::codecvt}} encapsulates conversion of character strings, including wide and multibyte, from one encoding to another. All file I/O operations performed through {{c/core|std::basic_fstream<CharT>}} use the {{c/core|std::codecvt<CharT, char, std::mbstate_t>}} facet of the locale imbued in the stream.
  
 
{{inheritance diagram/std-codecvt}}
 
{{inheritance diagram/std-codecvt}}
  
−
Four standalone (locale-independent) specializations are provided by the standard library:
+
===Specializations===
−
 
+
The standard library is guaranteed to provide the following specializations (they are {{rlp|locale|required to be implemented by any locale object}}):
 
{{dsc begin}}
 
{{dsc begin}}
−
{{dsc header | locale }}
+
{{dsc header|locale}}
−
{{dsc | {{c|std::codecvt<char, char, std::mbstate_t>}} | identity conversion }}
+
{{dsc|{{c/core|std::codecvt<char, char, std::mbstate_t>}}|identity conversion}}
−
{{dsc | {{c|std::codecvt<char16_t, char, std::mbstate_t>}} | conversion between UTF-16 and UTF-8 {{mark since c++11}}}}
+
{{dsc|{{c/core|std::codecvt<char16_t, char, std::mbstate_t>}}<br>{{mark life|since=c++11|deprecated=c++20}}|conversion between UTF-16 and UTF-8}}
−
{{dsc | {{c|std::codecvt<char32_t, char, std::mbstate_t>}} | conversion between UTF-32 and UTF-8 {{mark since c++11}}}}
+
{{dsc|{{c/core|std::codecvt<char16_t, char8_t, std::mbstate_t>}}<br>{{mark since c++20}}{{mark deprecated}}|conversion between UTF-16 and UTF-8}}
−
{{dsc | {{c|std::codecvt<wchar_t, char, std::mbstate_t>}} | conversion between the system's native wide and the single-byte narrow character sets}}
+
{{dsc|{{c/core|std::codecvt<char32_t, char, std::mbstate_t>}}<br>{{mark life|since=c++11|deprecated=c++20}}|conversion between UTF-32 and UTF-8}}
 +
{{dsc|{{c/core|std::codecvt<char32_t, char8_t, std::mbstate_t>}}<br>{{mark since c++20}}{{mark deprecated}}|conversion between UTF-32 and UTF-8}}
 +
{{dsc|{{c/core|std::codecvt<wchar_t, char, std::mbstate_t>}}|conversion between the system's native wide and the single-byte narrow character sets}}
 
{{dsc end}}
 
{{dsc end}}
−
 
−
In addition, every locale object constructed in a C++ program implements its own (locale-specific) versions of these four specializations.
 
  
 
===Member types===
 
===Member types===
 
{{dsc begin}}
 
{{dsc begin}}
−
{{dsc hitem | Member type | Definition}}
+
{{dsc hitem|Member name|Definition}}
−
{{dsc | {{tt|intern_type}} | {{tt|InternT}}}}
+
{{dsc|{{tt|intern_type}}|{{tt|InternT}}}}
−
{{dsc | {{tt|extern_type}} | {{tt|ExternT}}}}
+
{{dsc|{{tt|extern_type}}|{{tt|ExternT}}}}
−
{{dsc | {{tt|state_type}} | {{tt|State}}}}
+
{{dsc|{{tt|state_type}}|{{tt|StateT}}}}
 
{{dsc end}}
 
{{dsc end}}
  
 
===Member functions===
 
===Member functions===
 
{{dsc begin}}
 
{{dsc begin}}
−
{{dsc mem ctor | cpp/locale/codecvt/codecvt | constructs a new codecvt facet }}
+
{{dsc mem ctor|cpp/locale/codecvt/codecvt|constructs a new codecvt facet}}
−
{{dsc prot mem dtor | cpp/locale/codecvt/~codecvt | destructs a codecvt facet }}
+
{{dsc inc|cpp/locale/codecvt/dsc out}}
−
{{dsc inc | cpp/locale/codecvt/dsc out}}
+
{{dsc inc|cpp/locale/codecvt/dsc in}}
−
{{dsc inc | cpp/locale/codecvt/dsc in}}
+
{{dsc inc|cpp/locale/codecvt/dsc unshift}}
−
{{dsc inc | cpp/locale/codecvt/dsc unshift}}
+
{{dsc inc|cpp/locale/codecvt/dsc encoding}}
−
{{dsc inc | cpp/locale/codecvt/dsc encoding}}
+
{{dsc inc|cpp/locale/codecvt/dsc always_noconv}}
−
{{dsc inc | cpp/locale/codecvt/dsc always_noconv}}
+
{{dsc inc|cpp/locale/codecvt/dsc length}}
−
{{dsc inc | cpp/locale/codecvt/dsc length}}
+
{{dsc inc|cpp/locale/codecvt/dsc max_length}}
−
{{dsc inc | cpp/locale/codecvt/dsc max_length}}
+
 
{{dsc end}}
 
{{dsc end}}
  
 
===Member objects===
 
===Member objects===
 
{{dsc begin}}
 
{{dsc begin}}
−
{{dsc hitem | Member name | Type}}
+
{{dsc mem obj|nolink=true|{{dsc small|static std::locale::id}} id|''id'' of the locale}}
−
{{dsc | {{tt|id}} {{mark|static}} | {{lc|std::locale::id}} }}
+
 
{{dsc end}}
 
{{dsc end}}
  
 
===Protected member functions===
 
===Protected member functions===
 
{{dsc begin}}
 
{{dsc begin}}
−
{{dsc inc | cpp/locale/codecvt/dsc do_out}}
+
{{dsc prot mem dtor|cpp/locale/codecvt/~codecvt|destructs a codecvt facet}}
−
{{dsc inc | cpp/locale/codecvt/dsc do_in}}
+
{{dsc inc|cpp/locale/codecvt/dsc do_out}}
−
{{dsc inc | cpp/locale/codecvt/dsc do_unshift}}
+
{{dsc inc|cpp/locale/codecvt/dsc do_in}}
−
{{dsc inc | cpp/locale/codecvt/dsc do_encoding}}
+
{{dsc inc|cpp/locale/codecvt/dsc do_unshift}}
−
{{dsc inc | cpp/locale/codecvt/dsc do_always_noconv}}
+
{{dsc inc|cpp/locale/codecvt/dsc do_encoding}}
−
{{dsc inc | cpp/locale/codecvt/dsc do_length}}
+
{{dsc inc|cpp/locale/codecvt/dsc do_always_noconv}}
−
{{dsc inc | cpp/locale/codecvt/dsc do_max_length}}
+
{{dsc inc|cpp/locale/codecvt/dsc do_length}}
 +
{{dsc inc|cpp/locale/codecvt/dsc do_max_length}}
 
{{dsc end}}
 
{{dsc end}}
  
Line 69: Line 65:
  
 
===Example===
 
===Example===
−
 
 
{{example
 
{{example
−
| The following examples reads a UTF-8 file using a locale which implements UTF-8 conversion in codecvt<wchar_t, char, mbstate_t> and converts a UTF-8 string to UTF-16 using one of the standard specializations of std::codecvt
+
|The following examples reads a UTF-8 file using a locale which implements UTF-8 conversion in {{c/core|codecvt<wchar_t, char, std::mbstate_t>}} and converts a UTF-8 string to UTF-16 using one of the standard specializations of {{tt|std::codecvt}}.
−
| code=
+
|code=
 +
#include <codecvt>
 +
#include <cstdint>
 +
#include <fstream>
 +
#include <iomanip>
 
#include <iostream>
 
#include <iostream>
−
#include <fstream>
+
#include <locale>
 
#include <string>
 
#include <string>
−
#include <locale>
 
−
#include <iomanip>
 
−
#include <codecvt>
 
  
 
// utility wrapper to adapt locale-bound facets for wstring/wbuffer convert
 
// utility wrapper to adapt locale-bound facets for wstring/wbuffer convert
Line 84: Line 80:
 
struct deletable_facet : Facet
 
struct deletable_facet : Facet
 
{
 
{
−
     template<class ...Args>
+
     template<class... Args>
−
     deletable_facet(Args&& ...args) : Facet(std::forward<Args>(args)...) {}
+
     deletable_facet(Args&&... args) : Facet(std::forward<Args>(args)...) {}
 
     ~deletable_facet() {}
 
     ~deletable_facet() {}
 
};
 
};
Line 92: Line 88:
 
{
 
{
 
     // UTF-8 narrow multibyte encoding
 
     // UTF-8 narrow multibyte encoding
−
     std::string data = u8"z\u00df\u6c34\U0001f34c";
+
     std::string data = reinterpret_cast<const char*>(+u8"z\u00df\u6c34\U0001f34c");
−
                       // or u8"zß水🍌"
+
                       // or reinterpret_cast<const char*>(+u8"zß水🍌")
−
                       // or "\x7a\xc3\x9f\xe6\xb0\xb4\xf0\x9f\x8d\x8c";
+
                       // or "\x7a\xc3\x9f\xe6\xb0\xb4\xf0\x9f\x8d\x8c"
−
 
+
   
 
     std::ofstream("text.txt") << data;
 
     std::ofstream("text.txt") << data;
−
 
+
   
 
     // using system-supplied locale's codecvt facet
 
     // using system-supplied locale's codecvt facet
 
     std::wifstream fin("text.txt");
 
     std::wifstream fin("text.txt");
−
     // reading from wifstream will use codecvt<wchar_t, char, mbstate_t>
+
     // reading from wifstream will use codecvt<wchar_t, char, std::mbstate_t>
 
     // this locale's codecvt converts UTF-8 to UCS4 (on systems such as Linux)
 
     // this locale's codecvt converts UTF-8 to UCS4 (on systems such as Linux)
 
     fin.imbue(std::locale("en_US.UTF-8"));
 
     fin.imbue(std::locale("en_US.UTF-8"));
−
     std::cout << "The UTF-8 file contains the following UCS4 code points: \n";
+
     std::cout << "The UTF-8 file contains the following UCS4 code units:\n" << std::hex;
−
     for (wchar_t c; fin >> c; )
+
     for (wchar_t c; fin >> c;)
−
         std::cout << "U+" << std::hex << std::setw(4) << std::setfill('0') << c << '\n';
+
         std::cout << "U+" << std::setw(4) << std::setfill('0')
−
 
+
                  << static_cast<uint32_t>(c) << ' ';
 +
   
 
     // using standard (locale-independent) codecvt facet
 
     // using standard (locale-independent) codecvt facet
 
     std::wstring_convert<
 
     std::wstring_convert<
 
         deletable_facet<std::codecvt<char16_t, char, std::mbstate_t>>, char16_t> conv16;
 
         deletable_facet<std::codecvt<char16_t, char, std::mbstate_t>>, char16_t> conv16;
 
     std::u16string str16 = conv16.from_bytes(data);
 
     std::u16string str16 = conv16.from_bytes(data);
−
 
+
   
−
     std::cout << "The UTF-8 file contains the following UTF-16 code points: \n";
+
     std::cout << "\n\nThe UTF-8 file contains the following UTF-16 code units:\n"
 +
              << std::hex;
 
     for (char16_t c : str16)
 
     for (char16_t c : str16)
−
         std::cout << "U+" << std::hex << std::setw(4) << std::setfill('0') << c << '\n';
+
         std::cout << "U+" << std::setw(4) << std::setfill('0')
 +
                  << static_cast<uint16_t>(c) << ' ';
 +
    std::cout << '\n';
 
}
 
}
−
| output=
+
|output=
−
The UTF-8 file contains the following UCS4 code points:
+
The UTF-8 file contains the following UCS4 code units:
−
U+007a
+
U+007a U+00df U+6c34 U+1f34c  
−
U+00df
+
 
−
U+6c34
+
The UTF-8 file contains the following UTF-16 code units:
−
U+1f34c
+
U+007a U+00df U+6c34 U+d83c U+df4c  
−
The UTF-8 file contains the following UTF-16 code points:
+
−
U+007a
+
−
U+00df
+
−
U+6c34
+
−
U+d83c
+
−
U+df4c
+
 
}}
 
}}
 +
 +
===Defect reports===
 +
{{dr list begin}}
 +
{{dr list item|wg=lwg|dr=3767|std=C++20|before={{c/core|std::codecvt<char16_t, char8_t, std::mbstate_t>}} and<br>{{c/core|std::codecvt<char32_t, char8_t, std::mbstate_t>}} are locale-independent|after=deprecated them}}
 +
{{dr list end}}
  
 
===See also===
 
===See also===
 
{{cpp/locale/unicode string conversions}}
 
{{cpp/locale/unicode string conversions}}
 
{{dsc begin}}
 
{{dsc begin}}
−
{{dsc tclass | cpp/locale/codecvt_base | defines character conversion errors }}
+
{{dsc inc|cpp/locale/dsc codecvt_base}}
−
{{dsc tclass | cpp/locale/codecvt_byname | creates a codecvt facet for the named locale }}
+
{{dsc inc|cpp/locale/dsc codecvt_byname}}
−
{{dsc inc | cpp/locale/dsc codecvt_utf8}}
+
{{dsc inc|cpp/locale/dsc codecvt_utf8}}
−
{{dsc inc | cpp/locale/dsc codecvt_utf16}}
+
{{dsc inc|cpp/locale/dsc codecvt_utf16}}
−
{{dsc inc | cpp/locale/dsc codecvt_utf8_utf16}}
+
{{dsc inc|cpp/locale/dsc codecvt_utf8_utf16}}
 
{{dsc end}}
 
{{dsc end}}
  
−
[[de:cpp/locale/codecvt]]
+
{{langlinks|de|es|fr|it|ja|pt|ru|zh}}
−
[[es:cpp/locale/codecvt]]
+
−
[[fr:cpp/locale/codecvt]]
+
−
[[it:cpp/locale/codecvt]]
+
−
[[ja:cpp/locale/codecvt]]
+
−
[[pt:cpp/locale/codecvt]]
+
−
[[ru:cpp/locale/codecvt]]
+
−
[[zh:cpp/locale/codecvt]]
+

Latest revision as of 23:12, 27 June 2024

 
C++
 
Localization library
 
std::codecvt
 
Defined in header <locale>
template<

    class InternT,
    class ExternT,
    class StateT

> class codecvt;

Class template std::codecvt encapsulates conversion of character strings, including wide and multibyte, from one encoding to another. All file I/O operations performed through std::basic_fstream<CharT> use the std::codecvt<CharT, char, std::mbstate_t> facet of the locale imbued in the stream.

cpp/locale/codecvt basecpp/locale/locale/facetstd-codecvt-inheritance.svg

Inheritance diagram

Contents

[edit] Specializations

The standard library is guaranteed to provide the following specializations (they are required to be implemented by any locale object):

Defined in header <locale>
std::codecvt<char, char, std::mbstate_t> identity conversion
std::codecvt<char16_t, char, std::mbstate_t>
(since C++11)(deprecated in C++20) 		conversion between UTF-16 and UTF-8
std::codecvt<char16_t, char8_t, std::mbstate_t>
(since C++20)(deprecated) 		conversion between UTF-16 and UTF-8
std::codecvt<char32_t, char, std::mbstate_t>
(since C++11)(deprecated in C++20) 		conversion between UTF-32 and UTF-8
std::codecvt<char32_t, char8_t, std::mbstate_t>
(since C++20)(deprecated) 		conversion between UTF-32 and UTF-8
std::codecvt<wchar_t, char, std::mbstate_t> conversion between the system's native wide and the single-byte narrow character sets

[edit] Member types

Member name Definition
intern_type InternT
extern_type ExternT
state_type StateT

[edit] Member functions

 constructs a new codecvt facet
(public member function)
invokes do_out
(public member function) [edit]
invokes do_in
(public member function) [edit]
invokes do_unshift
(public member function) [edit]
invokes do_encoding
(public member function) [edit]
invokes do_always_noconv
(public member function) [edit]
invokes do_length
(public member function) [edit]
invokes do_max_length
(public member function) [edit]

[edit] Member objects

static std::locale::id id
 id of the locale
(public member object)

[edit] Protected member functions

 destructs a codecvt facet
(protected member function)
[virtual]
 converts a string from InternT to ExternT, such as when writing to file
(virtual protected member function) [edit]
[virtual]
 converts a string from ExternT to InternT, such as when reading from file
(virtual protected member function) [edit]
[virtual]
 generates the termination character sequence of ExternT characters for incomplete conversion
(virtual protected member function) [edit]
[virtual]
 returns the number of ExternT characters necessary to produce one InternT character, if constant
(virtual protected member function) [edit]
[virtual]
 tests if the facet encodes an identity conversion for all valid argument values
(virtual protected member function) [edit]
[virtual]
 calculates the length of the ExternT string that would be consumed by conversion into given InternT buffer
(virtual protected member function) [edit]
[virtual]
 returns the maximum number of ExternT characters that could be converted into a single InternT character
(virtual protected member function) [edit]

Inherited from std::codecvt_base

Member type Definition
enum result { ok, partial, error, noconv }; Unscoped enumeration type
Enumeration constant Definition
ok conversion was completed with no error
partial not all source characters were converted
error encountered an invalid character
noconv no conversion required, input and output types are the same

[edit] Example

The following examples reads a UTF-8 file using a locale which implements UTF-8 conversion in codecvt<wchar_t, char, std::mbstate_t> and converts a UTF-8 string to UTF-16 using one of the standard specializations of std::codecvt.

Run this code
#include <codecvt>
#include <cstdint>
#include <fstream>
#include <iomanip>
#include <iostream>
#include <locale>
#include <string>
 
// utility wrapper to adapt locale-bound facets for wstring/wbuffer convert
template<class Facet>
struct deletable_facet : Facet
{
    template<class... Args>
    deletable_facet(Args&&... args) : Facet(std::forward<Args>(args)...) {}
    ~deletable_facet() {}
};
 
int main()
{
    // UTF-8 narrow multibyte encoding
    std::string data = reinterpret_cast<const char*>(+u8"z\u00df\u6c34\U0001f34c");
                       // or reinterpret_cast<const char*>(+u8"zß水🍌")
                       // or "\x7a\xc3\x9f\xe6\xb0\xb4\xf0\x9f\x8d\x8c"
 
    std::ofstream("text.txt") << data;
 
    // using system-supplied locale's codecvt facet
    std::wifstream fin("text.txt");
    // reading from wifstream will use codecvt<wchar_t, char, std::mbstate_t>
    // this locale's codecvt converts UTF-8 to UCS4 (on systems such as Linux)
    fin.imbue(std::locale("en_US.UTF-8"));
    std::cout << "The UTF-8 file contains the following UCS4 code units:\n" << std::hex;
    for (wchar_t c; fin >> c;)
        std::cout << "U+" << std::setw(4) << std::setfill('0')
                  << static_cast<uint32_t>(c) << ' ';
 
    // using standard (locale-independent) codecvt facet
    std::wstring_convert<
        deletable_facet<std::codecvt<char16_t, char, std::mbstate_t>>, char16_t> conv16;
    std::u16string str16 = conv16.from_bytes(data);
 
    std::cout << "\n\nThe UTF-8 file contains the following UTF-16 code units:\n"
              << std::hex;
    for (char16_t c : str16)
        std::cout << "U+" << std::setw(4) << std::setfill('0')
                  << static_cast<uint16_t>(c) << ' ';
    std::cout << '\n';
}

Output: 

The UTF-8 file contains the following UCS4 code units:
U+007a U+00df U+6c34 U+1f34c 
 
The UTF-8 file contains the following UTF-16 code units:
U+007a U+00df U+6c34 U+d83c U+df4c

[edit] Defect reports

The following behavior-changing defect reports were applied retroactively to previously published C++ standards.

DR Applied to Behavior as published Correct behavior
LWG 3767 C++20 std::codecvt<char16_t, char8_t, std::mbstate_t> and
std::codecvt<char32_t, char8_t, std::mbstate_t> are locale-independent 		deprecated them

[edit] See also

Character
conversions 		locale-defined multibyte
(UTF-8, GB18030) 		UTF-8
 UTF-16
UTF-16 mbrtoc16 / c16rtomb (with C11's DR488)

codecvt<char16_t,char,mbstate_t>
codecvt_utf8_utf16<char16_t>
codecvt_utf8_utf16<char32_t>
codecvt_utf8_utf16<wchar_t>

N/A
UCS-2 c16rtomb (without C11's DR488) codecvt_utf8<char16_t> codecvt_utf16<char16_t>
UTF-32

mbrtoc32 / c32rtomb

codecvt<char32_t,char,mbstate_t>
codecvt_utf8<char32_t>

codecvt_utf16<char32_t>

system wchar_t:

UTF-32 (non-Windows)
UCS-2 (Windows)

mbsrtowcs / wcsrtombs
use_facet<codecvt
<wchar_t,char,mbstate_t>>(locale)

codecvt_utf8<wchar_t> codecvt_utf16<wchar_t>
defines character conversion errors
(class) [edit]
represents the system-supplied std::codecvt for the named locale
(class template) [edit]
(C++11)(deprecated in C++17)(removed in C++26)
 converts between UTF-8 and UCS-2/UCS-4
(class template) [edit]
(C++11)(deprecated in C++17)(removed in C++26)
 converts between UTF-16 and UCS-2/UCS-4
(class template) [edit]
(C++11)(deprecated in C++17)(removed in C++26)
 converts between UTF-8 and UTF-16
(class template) [edit]
Retrieved from "https://en.cppreference.com/mwiki/index.php?title=cpp/locale/codecvt&oldid=172924"